‘Installer-based AtScale’ to ‘Containerized AtScale’
This document is primarily for customers who are familiar with installer-based AtScale and considering moving to containerized AtScale. It explains the differences between the two architectures, management updates and development changes.
DevOps
RPM -> Containers
AtScale was completely re-architected to be a micro-serviced containerized platform. Installs are now Helm Chart orchestrated and managed by kubernetes.
The AtScale Platform
Platform Management
Organizations are Deprecated
An ‘organization’ in AtScale was a mechanism in AtScale which gave tenant-like capabilities. Each organization was administered (by org admin) as its own tenancy with global settings managed by the overall AtScale admin.
Advancements to metadata access and role-based access controls (RBAC) have made organization obsolete.
AtScale’s containerized platform now features an identity broker service, which is an enterprise grade Identity and Access Management solution. In addition to modernizing and expanding AtScale’s interoperability with Identity Providers (IdP), this new service provides access management as a comprehensive RBAC platform satisfying one of organizations’ primary functions: data management.
Access to the data models is now managed by the Git code repo. This enables the AtScale admin and peer Git admin to control which users and groups (teams) can access which repos including connections to data warehouses.
Git Repositories
The AtScale admin is now responsible for adding all Git repositories to the platform. Adding Git repos does not require authentication. An AtScale Design Center user will be able to access the repos via their token. Permissions to repos are managed on the Git platform.
License Management
By default, the AtScale entitlement service will attempt to validate the entered AtScale license to an internet accessible validation server. If the customer cannot support this “phone home” service, a billing summary will be required by the customer on a monthly basis.
Port Unification
AtScale’s architecture now unifies on standard web app ports. The AtScale UI no longer requires the 10500 port.
Data Modeling
Semantic Modeling Language
Semantic Modeling Language (SML) is a human readable YAML-based modeling approach. Data model constructs are broken out into discrete parts allowing for object-oriented construction by code or drag and drop. The AtScale developer can choose to work with the graphical canvas or the SML text editor or both.
While manually modifying project.xml wasn’t a recommended practice, AtScale customers have been known to edit this file. There were reasons to do so, which resulted in acknowledging that a code approach to modeling is necessary for today’s data engineers and DevOp professionals. SML makes it possible to automate modeling, integrate with code repositories, promote changes using code repos, make bulk modifications and more.
Project.xml does not technically go away, but it will no longer be accessible. Promotion of AtScale models will be done with SML. Project.xml should be considered a compiled artifact only for the AtScale engine to decipher.
Revamped UI
AtScale’s design center has been redesigned to incorporate new features and enhance the developer experience. Updates include:
- Maximizing real estate for development
- Semantic Modeling Language test editing
Design Time Projects are Deprecated
AtScale is evolving the concept of projects to repositories following industry CI/CD standards. Projects were designed to both organize and share: organize distinct use cases and share objects in a library. In practice, most AtScale customers had a one cube per project rule due to the lack of design time workflow isolation. In most cases, there was minimal sharing and projects became a folder for distinct use cases. Integrating with de facto code repo standards like Git, now introduces enterprise grade CI/CD capabilities. You can easily have one repo per AtScale data model, but now we expect our customers to have a shared repo with conformed dimensions, multiple developers working on common models, check in/out code, split off a branch and more.
CI/CD Changes
- No more Snapshots - Code is now saved locally and/or committed to code repo.
- No more auto-save - Now there is a save button.
- Undo now available - Revert option found in Source Control panel
- Diff now available - Compare changes prior to local or Git commiting
Code Repository Required
AtScale will now require the integration of a code repository in order to take advantage of standard CI/CD practices: repos, branches, multi-developer workspace, merge/diff and more. Code repos now manage design-time privileges rather than within the AtScale UI.
GitHub and GitLab are supported in the first release C2024.1, and other Git services will be considered for future releases.
The AtScale use of the term ‘cube’ fades away
AtScale has always presented data with business terms that fall under the umbrella of metrics and dimensions. This multidimensional data service has traditionally been known as ‘cubes’. Ensuring customers have performant and easy to understand data is not a new concept, OLAP providers have done this on small (relative to today’s data scales) datasets in the past. And again, those providers used the term ‘cube’. In order to evolve our platform and not create confusion, AtScale has moved away from the term ‘cube’ in deference to ‘model’.
Publish is now Deploy
In order to avoid confusion with newly integrated code repositories capability, AtScale has changed the command ‘publish’ to ‘deploy’.
Shared Library Deprecated
A dramatic shift in the AtScale CI/CD experience is the new approach in orchestrating shared model objects. The shared library within a design project is deprecated in favor of sharing model objects from a shared repository.
Data Model Querying
Postgres Interface
By default, AtScale will now use its Postgres interface for SQL queries. As such, SQL clients (like Tableau) will communicate with AtScale using a Postgres driver. Postgres is a de facto industry standard for database access, and AtScale will default to it for the query interface moving forward.
Tableau TDSs will also call for a Postgres driver by default. AtScale encourages our clients to migrate their applications over to this driver.
Excel Plug-in
AtScale now offers an Excel plug-in that allows Excel users to authenticate with AtScale while AtScale is configured with modern cloud-based identity providers.
PowerBI Token
A PowerBI developer seeking to access an AtScale data model will now use a more secure token fetchable from the AtScale UI. An AtScale admin, in turn, has the ability to swap an admin token from an admin console. A PowerBI user is not impacted at all, and their id will register in AtScale as the effective user.
Deployed Catalog
A set of deployed data models is now referred to as a catalog.
Security
Identity Management now managed by new Identity Broker
AtScale has traditionally supported multiple identity providers (IdP) and protocols with its in-house developed identity broker. In C2024.x, AtScale has augmented the platform with an enterprise identity broker. The new identity broker interfaces with the client’s IdP(s). The microservice architecture allows AtScale to leverage an off-the-shelf enterprise grade
IdP and its RBAC capabilities are extendable to other areas in the AtScale semantic layer platform.
Design time permissions now in Git
Repo access (read), commit (write) are managed in the Git platform (GitHub, GitLab, etc). The AtScale developer adds their Git token to the browser and any repos they have access to will display in the Repo browser.
Security Dimension now known as Row Security
While treated as a dimensional object, the “security dimension” was designed for row-level security. As such, AtScale renames the feature to “row security”. In SML, row security is not managed as a dimensional object.
API
Prior APIs supported in the RPM installer-based AtScale are included in the C2024.1 release with limited functionality and support. Subsequent releases starting with C2024.2 will deliver more features and functionality related to APIs.
Other
Telemetry Services
Previously, AtScale recorded actions in the internal Postgres database and wrote logs to local disk. The new Telemetry Services will offload logs to an object store (built-in or customer’s S3 location). The feature will offer a full set of dashboards to analyze the system behavior, resource thresholds, query optimization and more.
Dashboards will include monitoring for critical services, engine java virtual machine (JVM) performance and Postgres performance.