OpenMetadata Release 1.3

OpenMetadata offers the possibility to execute data quality tests directly from the UI, making it a powerful tool for everyone in the company. With close to 30 table and column tests, OpenMetadata makes it easy for everyone to own data quality.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*b828dVIB2rMBmubWZ0KeTQ.png"><figcaption>figure 1 — Creating a test from OpenMetadata UI</figcaption></figure>We understand that teams can use third-party tools, or have custom data quality rules. That is why we offer-third party integration with dbt and Great Expectations, and comprehensive API integration to ingest all your test results into the platform.
<figure><a href="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=small_rectangle"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*fUX9yyQ16MJvjWkevMKfHg.png"></a></figure>Now you may ask yourself, “What if I want to extend OpenMetadata tests, and define custom tests that can be set up and executed directly from the UI”? Then look no further. With OpenMetadata Data Quality Framework, you can define your data quality tests and make them available to all your users.
Disclaimer: You will need to work with the OpenMetadata API and write some Python code. You won’t need extensive knowledge of these two concepts; only basic familiarity with them will be enough.
<h4>Configure your Test Definition</h4>
The first step is to create your test definition. A test definition describes the different elements of your test (e.g., name, description, parameter, etc.). You will define this new test definition as shown below.
<pre>{ "description": "&lt;you test definition description&gt;", "entityType": "&lt;TABLE|COLUMN&gt;", "name": "&lt;your_test_name&gt;", "displayName": "&lt;your display name&gt;", "testPlatforms": ["OpenMetadata"], "supportedDataTypes": ["NUMBER|TEXT|CHAR"], "parameterDefinition": [ { "name": "&lt;name&gt;", "displayName": "&lt;display name&gt;", "description": "&lt;desription of the test&gt;", "optionValues": ["optionOne", "optionTwo"], }, { "name": "&lt;name&gt;", "displayName": "&lt;display name&gt;", "description": "&lt;desription of the test&gt;", "optionValues": ["optionOne", "optionTwo"], } ] }</pre>
Let’s go over each element one by one:
<ol>
<li>description: this is the description of your test. It will provide your users with more information and details</li>
<li>entityType: set TABLE if you are defining a table-level test or COLUMN if you are defining a column-level test</li>
<li>name: the name of your test definition. It should be unique across all test definitions</li>
<li>displayName: the name of your test definition as it will appear in OpenMetadata UI</li>
<li>testPlatforms: this is an array of test platforms. To make your test executable through OpenMetadata, this should be set to OpenMetadata</li>
<li>supportedDataTypes: (optional) pass a list of supported data types for your test definition. This will limit the ability for users to create the test only against columns with supported data types. The complete list of data type values can be found <a href="https://github.com/open-metadata/OpenMetadata/blob/222a8f898451f6dc7dafc7bfda232d91c21209cf/openmetadata-spec/src/main/resources/json/schema/entity/data/table.json#L74-L147">here</a>.</li>
<li>parameterDefinition: a list of parameters that will be used to configure a test name: the name of your parameter displayName: the display name of your parameter as it will appear in the UI description: the description for your parameter. This will be used to populate the tooltip in the UI optionValues: If you want to make your parameter value a selectable value from a dropdown menu</li>
</ol>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*DLYdtcS7iwkX3CU-GWLQuQ.png"></figure><h4>Creating your Test Definition in OpenMetadata</h4>
<h4>Create your Test Definition in OpenMetadata</h4>
Once you have defined your test definition using the JSON template above, you must make the test available in OpenMetadata. Making it available in OpenMetadata will make your test usable through the UI. Users can navigate to the test creation page and select it from the dropdown menu.
For this step, you will need a way to send an API POST request to your OpenMetadata server endpoint/api/v1/dataQuality/testDefinitions. For this example, we’ll use curl (more info about curl<a href="https://curl.se/docs/"> here</a>).
Test Definition
<pre>{ "description": "Compute entropy for a given column", "displayName": "Column Entropy To Be Between", "entityType": "COLUMN", "name": "columnEntropyToBeBetween", "testPlatforms": ["OpenMetadata"], "parameterDefinition": [ { "name": "minEntropy", "displayName": "Min", "dataType": "INT", "required": true, "description": "minimum entropy" }, { "name": "maxEntropy", "displayName": "Max", "dataType": "INT", "required": true, "description": "maximum entropy" } ] }</pre>
curl Request
<pre>curl --location --request POST '&lt;yourServerHost&gt;/api/v1/dataQuality/testDefinitions' \ --header 'Authorization: Bearer &lt;yourJWTToken&gt;' \ --header 'Content-Type: application/json' \ --data-raw '{ "description": "Compute entropy for a given column", "displayName": "Column Entropy To Be Between", "entityType": "COLUMN", "name": "columnEntropyToBeBetween", "testPlatforms": ["OpenMetadata"], "parameterDefinition": [ { "name": "minEntropy", "displayName": "Min", "dataType": "INT", "required": true, "description": "minimum entropy" }, { "name": "maxEntropy", "displayName": "Max", "dataType": "INT", "required": true, "description": "maximum entropy" } ] }'</pre>
You will need to replace &lt;yourServerHost&gt; with the URL where the OpenMetadata server is hosted (e.g., http://localhost:8558 if you are working with a local installation). You will also need to replace &lt;yourJWTToken&gt; with your personal JWT Token (i.e., go to settings &gt; bot to find your JWT Token or generate one).
<figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*rBf7hVbPqx9-ZSqNR9ZiUQ.png"><figcaption>figure 2 — Our custom entropy test in OpenMetadata UI</figcaption></figure><h4>Define the Execution Logic</h4>
Now that we have created our new test definition, we’ll need to tell OpenMetadata how we want to process and execute this test. To accomplish this, we can leverage OpenMetadata Data Quality Framework.
Step 1 — Create a Python Package
The first step is to create a new Python package. This will allow you to version control your custom tests and add unit tests. As we’ll be leveraging Python namesapce package, our package needs to have a specific structure.
<ul>
<li>For a column level test: &lt;yourFolderName&gt;/metadata/data_quality/validations/columns/&lt;sqlalchemy|pandas&gt;/yourTestName.py</li>
<li>&lt;yourFolderName&gt;/metadata/data_quality/validations/table/&lt;sqlalchemy|pandas&gt;/yourTestName.py</li>
</ul>
Use pandas if you are writing a test that will be used for data lake connectors; otherwise, use sqlalchemy for all other database connectors
For our column entropy test, our folder structure will look like this
<pre>folderName/ ├── setup.py └── metadata/ ├── __init__.py └── data_quality/ ├── __init__.py └── validations/ ├── __init__.py └── column/ ├── __init__.py └── sqlalchemy/ ├── __init__.py └── columnEntropyToBeBetween.py</pre>
Important: note how our test file name matches the above test definition name. This is important to respect.
Step 2 — setup.py and __init__.py files
For your setup.py file, you will, at minimum, need to call the setup function calling find_packages() for the packages argument.
For our entropy test, we have added the requirements needed to execute our test (e.g., scipy )
<pre>from setuptools import find_packages, setup base_requirements = ["scipy"] setup(name='custom-tests', version='0.0.1', description='OpenMetadata Custom Tests', packages=find_packages(), install_requires=list(base_requirements) )</pre>
Finally, you must add the line below to all your __init__.py files. This tells Python that we are working with namespace packages (this is at the core of OpenMetadata flexible Data Quality Framework).
<pre>__path__ = __import__('pkgutil').extend_path(__path__, __name__)</pre>
Step 3 — Defining your Test Execution Logic
One of the last two missing pieces to our puzzle is adding the logic to execute our test. You will need to create a new class name &lt;YourTestName&gt;Validator that will inherit from BaseTestValidator.
You will then need to implement the run_validation method where your core logic is being executed.
<pre>from itertools import chain from sqlalchemy import Column, inspect from scipy import stats from metadata.data_quality.validations.base_test_handler import BaseTestValidator from metadata.generated.schema.tests.basic import TestCaseResult, TestResultValue from metadata.utils.entity_link import get_decoded_column class ColumnEntropyToBeBetweenValidator(BaseTestValidator): """Implements custom test validator for OpenMetadata. Args: BaseTestValidator (_type_): inherits from BaseTestValidator """ def _get_column_name(self) -&gt; Column: """get column name from the test case entity link Returns: Column: column """ column = get_decoded_column(self.test_case.entityLink.__root__) columns = inspect(self.runner.table).c column_obj = next( (col for col in columns if col.name == column), None, ) if column_obj is None: raise ValueError(f"Cannot find column {column}") return column_obj def run_validation(self) -&gt; TestCaseResult: """Run test validation""" rows = self.runner.select_all_from_table(self._get_column_name()) list_of_values = list(chain.from_iterable(rows)) entropy = stats.entropy(list_of_values) min_bound = self.get_min_bound("minEntropy") max_bound = self.get_max_bound("maxEntropy") return self.get_test_case_result_object( self.execution_date, self.get_test_case_status(min_bound &lt;= entropy &lt;= max_bound), f"Found entropy={entropy} vs. the expected min={min_bound} and max={max_bound}].", [TestResultValue(name="entropy", value=str(entropy))], )</pre>
For our entropy test above, we created our new class ColumnEntropyToBeBetweenValidator, which inherits from BaseTestValidator. We then implemented run_validation, fetching the rows from our table column with self.runner.select_all_from_tabl(self._get_column_name()). self._get_column_name() fetches the column for which our test was created. We then compute the entropy for the column stats.entropy(list_of_values). Finally, we fetch the min and max bounds that were set self.get_min_bound("minEntropy") — note how the argument we pass to get_&lt;min|max&gt;_bound corresponds to the name of our test definition. Finally, we call self.get_test_case_result_object(...) which will return the status of our test alongside other information.
Step 4 — Install your package in your environment
Finally, you must install your package where your OpenMetadata ingestion is running using pip install &lt;...&gt;.
If you enjoy OpenMetadata and want to contribute new tests to the community, you can head to our <a href="https://github.com/open-metadata/OpenMetadata">GitHub Repo</a>, browse the <a href="https://github.com/open-metadata/OpenMetadata/tree/main/ingestion/src/metadata/data_quality">Data Quality code</a>, and open a new PR. 😊
<figure><a href="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=large_rectangle"><img alt="" src="https://cdn-images-1.medium.com/max/700/1*p3bWeQSUb_qBKGlariCwyQ.png"></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=385ba2d8eaf" width="1" height="1" alt=""><hr>
<a href="https://blog.open-metadata.org/leveraging-the-power-of-openmetadata-data-quality-framework-385ba2d8eaf">Leveraging the Power of OpenMetadata Data Quality Framework</a> was originally published in <a href="https://blog.open-metadata.org/">OpenMetadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Leveraging the Power of OpenMetadata Data Quality Framework

OpenMetadata Release 1.2

<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ioMhIqAhguFxd1YxKSRgGA.png"></figure>Data quality and reliability are two top priorities for organizations. Many organizations have made considerable investments in Data Quality initiatives. With the current crop of tools, success has been limited due to tool disconnect, lack of collaboration, and accessibility to a broader audience.
We have a different take on how data quality should be done. Our approach has been to start with metadata. Gathering metadata based on a <a href="https://docs.open-metadata.org/v1.2.x/main-concepts/metadata-standard/schemas">metadata standard</a> allows us to contextualize data in a centralized place and to develop applications and automation on top of this metadata standard — one of which is our data quality solution.
<figure><a href="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=small_rectangle"><img alt="" src="https://cdn-images-1.medium.com/max/720/0*T0_GHaaxmJ_NWU4W.png"></a></figure>In this blog, we’ll explore how <a href="https://open-metadata.org/">OpenMetadata</a> natively supports data quality that is key to many advantages — lowered cost, improved productivity, better <a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-collaboration">collaboration</a>, and many more. Let’s dive deep into some of these advantages to show why the approach taken by OpenMetadata is superior. We will use data validation tools like <a href="https://greatexpectations.io/">Great Expectations</a> and <a href="https://soda.io/">Soda</a> to contrast with <a href="https://open-metadata.org/">OpenMetadata</a>.
<h3>No need for yet another tool for data quality</h3>
With <a href="https://open-metadata.org/">OpenMetadata</a>, you don’t need a separate standalone system for data quality. The data quality validations are run as data workflows similar to workflows like ETL. This reduces the cost of buying yet another tool in the already fragmented data architecture and the operational costs of installing it, integrating it with various databases with various access control permissions, and managing another service.
<h3>Efficient creation and running of validation</h3>
The diagram below shows the steps for creating and running validation:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/984/0*EHfpglAmmQYfGB_O"></figure><a href="https://open-metadata.org/">OpenMetadata</a> reuses the existing metadata, and data source connections are already set up. For other tools, such as <a href="https://greatexpectations.io/">great expectations</a> or <a href="https://www.soda.io/">Soda</a> a connection to the data sources must be set up, and a duplicate copy of metadata must be created even before writing tests. Creating a test is simple in <a href="https://open-metadata.org/">OpenMetadata</a>; it requires only two steps and a few clicks. You can have a suite of assertions running against your table in no time. Writing validations for a table can be reduced from hours to minutes.
<h3>Rich data context makes writing validations easy</h3>
<a href="https://open-metadata.org/">OpenMetadata</a> provides rich data profile visualization that helps you identify data quality issues quickly and write validations to detect them, as shown below:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yo_xt7Np-ivaqBug"></figure>If you see a column with many unexpected null values, you can quickly add a test to validate for null values. If you see a column like total orders having unexpected negative values, you can add a test to check for a valid range.
<h3>No code tests to democratize data quality</h3>
For <a href="https://greatexpectations.io/">great expectations</a> and <a href="https://www.soda.io/">Soda</a>, writing tests involves installing Python packages, writing tests in YAML or Python, and running them using schedulers. This makes these tools only accessible to technical users. <a href="https://open-metadata.org/">OpenMetadata</a> provides simple no code and intuitive UI to allow even business users to write tests easily to democratize data quality. Suppose an analyst or a data scientist is making assumptions about the data (non-null, a range of values, cardinality, etc.); they can contribute a test to ensure those assumptions are validated. This helps build a culture where data quality is not just a responsibility of some technical users but a shared responsibility of the organization.
With the current data quality tools, there is no central place to store the tests and collect the results. You will also need to deploy and maintain a separate user interface to visualize your test results separately per project. Oftentimes, different teams write the same validations in siloes resulting in needless duplication of tests and unnecessary compute costs. All this can be avoided with the central metadata repository for all the tests and results with OpenMetadata.
<h3>Faster debugging and resolution of data quality issues</h3>
Writing validation is just the first step. These tests often fail for various reasons. Currently, debugging data quality issues requires a user to jump from tool to tool to understand lineage, schema changes, code changes, etc., as the data quality tools lack the complete context. Fixing the issue requires identifying owners and going from team to team, which can take multiple days of firefighting.
<a href="https://docs.open-metadata.org/">Central metadata</a> provides one place to get the complete picture of your data instead of piecing it together by jumping across the tools. When a validation fails, all the concerned users are alerted about what is failing. The end-to-end lineage is provided to help in impact analysis to identify where the failure originated. All the changes to data and ETL pipelines are versioned, which helps identify how a change might have introduced the issue. Users can easily identify the owner of the data assets to identify who should fix the issue, start a conversation directly inside <a href="https://open-metadata.org/">OpenMetadata</a>, and collaborate on resolving the issue. The complete context of what failed, where it failed, how it failed, and who is responsible for fixing it helps save countless hours in debugging and resolving data quality issues.
Communication is key when there are data quality issues. With <a href="https://open-metadata.org/">OpenMetadata</a>, you can also quickly implement a resolution workflow. Test failures will be marked as “New” in the entity data quality dashboard, informing users of ongoing failures. The status can be changed to “Ack” for acknowledgment to let users of the table know the failure is being investigated. Once resolved, users can edit the status to resolved and choose a failure reason that best explains the incident.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vTswLdOTog_zVXkD"></figure><h3>Organizational Visibility to Data Quality</h3>
In <a href="https://open-metadata.org/">OpenMetadata</a>, you will also find a data quality health dashboard, allowing you to understand the overall health of your quality assertions. This view is available at the data asset and the organization level. The dashboard shows how the organization is doing with data quality and where improvements are needed.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yARoIeOMx52-Igva"></figure><h3>Extensibility for power users</h3>
The native tests OpenMetadata offers cover a wide range of data quality dimensions. It provides assertions covering Accuracy, Completeness, Timeliness, Relevance, Uniqueness, and Validity. It enables teams from across the organization to collaborate on data quality definition to test the business standard of data sets and the engineering ones.
<a href="https://open-metadata.org/">OpenMetadata</a> goes one step further by making the data quality tool easily extensible. For users with SQL knowledge, our custom SQL test enables users to implement testing logic that is specific to their use case.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/969/0*A1xusP_lN0EsOeLK"></figure>Users with Python knowledge can easily extend the <a href="https://open-metadata.org/">OpenMetadata</a> quality framework. You can create and define your own data quality assertions, install them inside <a href="https://open-metadata.org/">OpenMetadata</a>, and have your users configure them and run them directly from the OpenMetadata UI — more details here. With its extensive SDKs and APIs, it is easy to integrate <a href="https://open-metadata.org/">OpenMetadata</a> tests with your data processing pipeline. You can define your tests in <a href="https://open-metadata.org/">OpenMetadata</a> and execute them as a step of your ETL/ELT process — more details <a href="https://blog.open-metadata.org/leveraging-the-power-of-openmetadata-data-quality-framework-385ba2d8eaf">here</a>.
While data quality programs have become strategic for organizations, few tools have approached it as such. Focused primarily on validation, requiring technical expertise, or treating metadata as a by-product, they have fallen short in solving the multiple challenges of data quality programs.
With its metadata standard first approach and easy-to-use and extensible data quality tool, <a href="https://open-metadata.org/">OpenMetadata</a> is the solution that can help organizations scale and foster innovation with reliable data at their fingertips.
<h3>Feature Comparison</h3>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*YRRm9X4cSmAFa-epq03g5A.png"></figure><h3>Data Quality , How to Guides</h3>
Please refer to our <a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-quality-profiler">documentation about Data Quality , Profiler</a> to learn more about the features in <a href="https://open-metadata.org/">OpenMetadata</a>
<h3>Road Ahead</h3>
We have many amazing features related to Data Quality in our roadmap:
Anomaly Detection — Automated Anomaly detection of Freshness, Volume, Completeness, and Schema change detection.
Suggest Tests for Tables — OpenMetadata will suggest tests based on Tables and Columns using Machine Learning.
Data Diff — Show the differences between tables across different database services.
Data SLAs — Data SLAs to provide guarantees around data assets.
<figure><a href="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=large_rectangle"><img alt="" src="https://cdn-images-1.medium.com/max/720/0*47MNbpPQEQIMKsvI.png"></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=1c4e7d329364" width="1" height="1" alt=""><hr>
<a href="https://blog.open-metadata.org/simple-easy-and-efficient-data-quality-with-openmetadata-1c4e7d329364">Simple, Easy, and Efficient Data Quality with OpenMetadata</a> was originally published in <a href="https://blog.open-metadata.org/">OpenMetadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Simple, Easy, and Efficient Data Quality with OpenMetadata

<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tB1R-iQuSWMtVExWb-_NQw.png"></figure><h4>UI Overhaul, New Connectors, Improved Lineage Parsing, PII Masking, and lots more</h4>
Written By: OpenMetadata Team
Less than two months have passed since our exciting <a href="https://blog.open-metadata.org/openmetadata-release-1-0-the-journey-so-far-6b6d9319211e">OpenMetadata 1.0 Release</a>, and we’re thrilled to announce the completion of Release 1.1 already! The OpenMetadata community thrives on pushing our limits; this latest release is a testament to it. Prepare to be amazed as we unveil a complete UI overhaul, meticulously designed to elevate the user experience across the entire platform. But that’s not all! We’ve also introduced four new connectors, implemented advanced PII masking, and significantly enhanced lineage parsing capabilities, just to name a few of the numerous features we’ve packed into this release. Stay tuned for an exceptional OpenMetadata experience like never before!
The OpenMetadata Community recently hosted a Webinar showcasing <a href="https://www.youtube.com/watch?v=i7DhG_gZMmE&amp;ab_channel=OpenMetadata">How the ingestion Framework Works</a>. The bottom line: No Airflow? No Problem! While the one-click deploy and manage workflows in UI might not be there, you can still run the exact same processes from — literally — anywhere. In the webinar, we <a href="https://github.com/open-metadata/openmetadata-demo/tree/main/ingestion-github-actions">demo</a> ingestion workflows executed from GitHub Actions.
<figure><a href="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=small_rectangle"><img alt="" src="https://cdn-images-1.medium.com/max/728/1*M_tTyWDVeLK72TX0_DklIg.png"></a></figure>In the upcoming 1.2 Release of OpenMetadata, we are thrilled to introduce exclusive new features specifically tailored for Collate SaaS. You can review <a href="https://www.getcollate.io/roadmap">Collate’s roadmap here</a> and be as excited as we are 🚀
<h3>Community Updates</h3>
Thanks to the incredible OpenMetadata Community, our growth and activity have skyrocketed. Slack is buzzing with constant engagement, and we truly appreciate your code contributions, feedback, and feature requests. Our <a href="https://www.youtube.com/watch?v=i7DhG_gZMmE&amp;ab_channel=OpenMetadata">webinars</a> are attracting more attendees, and the June community meeting was extra special, thanks to our first <a href="https://www.youtube.com/watch?v=AIGlN4ZemSY&amp;ab_channel=OpenMetadata">Community Spotlight</a>: <a href="https://www.linkedin.com/in/ga%C3%A9tan-soulas-99279b24/">Gaétan Soulas</a> from <a href="http://solocal.com/">Solocal</a>!
We are excited about our soaring community numbers!
<ul>
<li>Crossed 2400+ GitHub stars (+200 stars since the previous release)</li>
<li>The Slack community reached 3200+ members (+500 since the previous release)</li>
<li>
168 Open-source GitHub developers (+8 since the previous release)</li>
<li>Merged 526 Commits into the 1.1 Release
</li>
</ul>
<h3>OpenMetadata 1.1 Release Highlights</h3>
<h4>UI Overhaul</h4>
This release marks a significant milestone for the OpenMetadata platform, bringing many UI changes that are among the most substantial since the start of the project in 2021.
Our primary focus is to simplify the overall experience for users while building upon our already exceptional UI. We are incredibly excited to share these changes with you as they further enhance the platform’s discovery, collaboration, and data quality experience.
<h4>Refined Landing Page</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XiXYEKI9FrmsNBqmwvo70A.gif"></figure>One notable improvement is the revamped Landing Page, which aims to ensure users from diverse roles can navigate and explore metadata effortlessly. From the start, we want every user to have a seamless and enjoyable interaction with their data.
We will continue to refine and evolve the Landing Page components, allowing organizations to customize the user experience for different Personas. This flexibility enables highlighting specific areas of interest such as discovery, governance, management, and more.
We are just scratching the surface of what’s to come, so stay tuned for more updates.
<h4>Simplified Explore View for Effortless Discovery</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wz2uhFXWQlJHJkBU9hk1Mg.gif"></figure>The search experience has been enhanced with an improved Asset Details section and a simplified filtering experience with a unique top bar. Before, the explore page was packed with tons of information, and users had filtering options in different places on the screen.
With a centralized approach for filtering, we want users to be able to really focus on what’s important: the data.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Gw6ZbuU-NdRAehdgeUVVQ.gif"></figure><h4>Improved Lineage View</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LYab3eZKngnj8Edzpf6OXg.gif"></figure>Data Lineage has been consistently enhanced in every release. In the current release, we have added support for column pagination and filtering. This will make the exploration of tables with a large number of columns simpler and faster.
<h4>Data Quality Redesign</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PQLLMUYrznRubdTbP4jz0w.gif"></figure><a href="https://github.com/open-metadata/OpenMetadata/issues/11592">Data Quality Tests</a> have been redesigned for an improved end-user experience. You no longer need to select a test suite when adding tests to a table. This helps organize and schedule all the tests related to a table in a single place, which also helps prevent unnecessary duplication of tests.
The role of Test Suites is now to help users create their own control panels of the tests they’d like to monitor, even if they belong to multiple tables.
With these changes, we still maintain the flexibility of grouping tests based on your use case or domain while simplifying the creation and scheduling flows.
<h4>Test Case Failures Resolution</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WhoW9LuNGlcWQLxGT4NHmA.gif"></figure>What do we do when a test fails? As an end user of a table, how do I know if an error is being reviewed or already solved? Communication is vital when managing failures or outages, and OpenMetadata now helps streamline this process to ensure no information is lost.
Adding the Resolution field to failed test cases will allow users to:
<ol>
<li>Acknowledge the failure once the situation is being analyzed,</li>
<li>Update the status to Resolved with the following reasons: Duplicates, False Positive, Missing Data, Out of Bounds, or. Other, if anything other than the above, applies.</li>
<li>Finally, provide any further explanations.</li>
</ol>
We will introduce more features and options on this Resolution flow. To allow users to explore the already resolved tests that could help identify ongoing issues.
<h4>Improved SQL Parsing</h4>
Open source thrives when communities grow together. <a href="https://github.com/reata/sqllineage">SQLLineage</a> and <a href="https://github.com/sqlfluff">SQLFluff</a> are the foundations of OpenMetadata’s lineage parsing capabilities. When ingesting lineage from database sources, we get the queries executed on that instance from the query history tables. These queries are then parsed through SQLFluff and SQLLineage to extract the involved tables and their relationship.
In this release, we have continued our efforts to improve the SQL parsing by directly contributing to both communities. We will continue to improve the open-source ecosystem release after release, with your <a href="https://github.com/open-metadata/OpenMetadata/issues/7427">help</a> in identifying those queries that still need support.
<h4>Connectors</h4>
This release brings out a long list of new features and improvements for existing connectors. On top of that, three new connectors have been added, giving us a total of 66 connectors supported by OpenMetadata.
Here are the highlights of this release:
<ul>
<li>New <a href="https://absaoss.github.io/spline/">Spline</a> Connector to extract metadata and lineage from Spark jobs. Regardless of where the Spark execution happens, if you have configured the Spline Agent, we can send Spark metadata to OpenMetadata.</li>
<li>New <a href="https://www.sap.com/products/technology-platform/hana/what-is-sap-hana.html">SAP Hana</a> Connector, our first integration to the SAP ecosystem.</li>
<li>New <a href="https://www.mongodb.com/">MongoDB</a> Connector, extracting Collections as Tables.</li>
<li>New Oracle lineage and usage workflows based on the query history.</li>
<li>Added support for <a href="https://www.databricks.com/product/unity-catalog">Databricks Unity Catalog</a> for metadata and lineage extraction. If your Databricks instance supports the Unity Catalog, you can enable it in the Connection Details section to use this metadata extraction method instead of getting metadata out of the metastore and history APIs.</li>
<li>IAM role-based authentication for MySQL and Postgres RDS databases.</li>
</ul>
<h4>PII Masking</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QvVjKNNDfr6Dwe2F12M0cg.gif"></figure>Crucial data must always be secure in the hands of the concerned stakeholders.
That is why we are introducing PII Masking capabilities to the platform. Let’s break down the scenarios:
We can tag the table or its columns as PII Sensitive for Tables. We will mask the sample data, profiler metrics, and test cases if tagged. The only users that can see the real value are the owners of the tables or admins. This feature goes perfectly with the Auto-Tagging capability introduced in 1.0!
We can apply the same tagging logic for Topics to mask the sample data completely. Finally, Queries will also be masked for non-owners.
In the 1.2 release, we will iterate on this behavior to increase flexibility based on roles and policies to manage who can view sensitive information.
<h4>FQN Hashing: Supports Longer Names!</h4>
While this is an internal detail, it took weeks of planning and implementation, and we are excited to share the same with you!
As you already know, OpenMetadata is based on a metadata standard powered by JSON Schema. When we bring metadata into the platform, it transforms into this standard and is safely stored in a database (for the curious, more information on the platform’s high-level design <a href="https://docs.open-metadata.org/v1.0.0/main-concepts/high-level-design">here</a>).
Since we want the API to be blazing fast, how we define and manage indexes in the db was a limiting factor on how long the entity names in OpenMetadata could be. In this release, we have improved how we store the data. By storing the hashed names, we can now support longer names in the application without hitting the limits of index sizes.
This change is especially important for tables that come from the Datalake connector (named after the path) or containers from the Storage Services, which can now be nested further.
This is just another example of how we continuously expand the metadata standard and APIs to fit more use cases while retaining architectural simplicity and high <a href="https://blog.open-metadata.org/openmetadata-benchmarking-3120782f8a12">performance</a>.
<h3>Planned for Release 1.2</h3>
We have stated the highlights of our current release above, but more exciting updates are in store for the 1.2 Release! We’ll bring support for Domains, expand the Custom Properties for all entities, and further improve the Glossaries with approval workflows, among many other features! You can check the roadmap <a href="https://docs.open-metadata.org/v1.0.0/releases/roadmap">here</a>.
Moreover, the Collate team will continue to expand the platform’s capabilities with features built for the SaaS version of OpenMetadata. Review the <a href="https://www.getcollate.io/roadmap">Collate roadmap here</a>, and <a href="https://www.getcollate.io/contact">let us know your thoughts</a>!
<figure><a href="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=large_rectangle"><img alt="" src="https://cdn-images-1.medium.com/max/728/1*B5kZqafplTME5JQ_COWGwA.png"></a></figure><h3>Thanks to our Contributors</h3>
We are grateful for the active participation of the OpenMetadata community in making this open-source project successful. It is good to see code contributions from the Community as well as the amazing feedback we have been receiving. Thank you so much for exploring the Sandbox and using OpenMetadata in your organizations. The GitHub issues are a validation of the increased adoption of OpenMetadata in diverse industry verticals.
Thank you, <a href="https://github.com/schlameel">Schlameel</a>, for the code contribution, i.e., adding methods to update Glossary and GlossaryTerm in Python SDK. Thanks to the following users for providing feedback on GitHub that made it to the 1.1 release: <a href="https://github.com/AlekseevVadim">Alekseev Vadim</a>, <a href="https://github.com/allenhaozi">Allen Haozi</a>, <a href="https://github.com/Artemnest">Artemnest</a>, <a href="https://github.com/bmy-ashampoo">Benjamin Meyer</a>, <a href="https://github.com/bvujanic">Bojan Vujanic</a>, <a href="https://github.com/cutz">Chris Utz</a>, <a href="https://github.com/cmedinasoriano">Christian Medina</a>, <a href="https://github.com/cristiancalugaru">Cristian Calugaru</a>, <a href="https://github.com/Daandamhuis">Daan Damhuis</a>, <a href="https://github.com/deepaktripathi1997">Deepak Tripathi</a>, <a href="https://github.com/DenisKirichenko24">Denis Kirichenko</a>, <a href="https://github.com/dhu-gehang">Dhu-Gehang</a>, <a href="https://github.com/dim1tri">Dimitri</a>, <a href="https://github.com/DovileKr">DovileKr</a>, <a href="https://github.com/eeshan-deshpande">Eeshan Deshpande</a>, <a href="https://github.com/felipecgonc">Felipe Caracciolo Gonçalves</a>, <a href="https://github.com/geoHeil">Georg Heiler</a>, <a href="https://github.com/giacobbino">Giacomo</a>, <a href="https://github.com/CorsettiS">Giovanni Corsetti</a>, <a href="https://github.com/gnomolio">Gnomolio</a>, <a href="https://github.com/gzusgw">Guangwei</a>, <a href="https://github.com/quguangyu0321">Guangyu Qu</a>, <a href="https://github.com/haithem-souala">Haithem Souala</a>, <a href="https://github.com/HesamoddinMonfared">Hesamoddin Monfared</a>, <a href="https://github.com/oynek">Jan-Niklas Mühlenbrock</a>, Jan Picha, <a href="https://github.com/JonathanBonnaud">Jonathan Bonnaud</a>, <a href="https://github.com/joosepvaikma">Joosep Vaikma</a>, <a href="https://github.com/jromandv">JRomandv</a>, <a href="https://github.com/kputland">Karl Putland</a>, <a href="https://github.com/keatmin">Keat Min</a>, <a href="https://github.com/KevinParijs">Kevin Parijs</a>, <a href="https://github.com/lailapatel">Laila Patel</a>, <a href="https://github.com/larspettermadsstuen">Larspetter Madsstuen</a>, <a href="https://github.com/learn-learn4">learn-learn4</a>, <a href="https://github.com/trillhaa">Martin Trillhaas</a>, <a href="https://github.com/martsec">Martsec</a>, <a href="https://github.com/mmalta96">Matheus Malta</a>, <a href="https://github.com/mhornst">Miki Horn</a>, <a href="https://github.com/mithmatt">Mith Matt</a>, <a href="https://github.com/NoelAramis">Noel Aramis</a>, <a href="https://github.com/ph-eip">Ph-eip</a>, <a href="https://github.com/cyclonstep">Presteniko Septi Rahadian</a>, <a href="https://github.com/17rahulsharma">Rahul Sharma</a>, <a href="https://github.com/raja-mudhireddy">Raja Mudhireddy</a>, <a href="https://github.com/raphaelsolarski">Raphael Solarski</a>, <a href="https://github.com/ruffury">Rufat</a>, <a href="https://github.com/SabineGl">Sabine GI</a>, <a href="https://github.com/Sai7656">Sai</a>, <a href="https://github.com/sapirhi">SapirHi</a>, <a href="https://github.com/sebaap">Sebastian Arriagada</a>, <a href="https://github.com/sevdegueldre">Sevde Gueldre</a>, <a href="https://github.com/SharpLu">Sharp Lu</a>, <a href="https://github.com/stephanesol">Stéphane Sol</a>, <a href="https://github.com/tjpegram">Tyler Pegram</a>, <a href="https://github.com/yosank">Yossi Ankri</a> <a href="https://github.com/yu-iskw">Yu Ishikawa</a>, <a href="https://github.com/yuqi-sun">Yuqi Sun</a>, and <a href="https://github.com/zach-all">Zach All</a>.
Please get in touch with us on <a href="https://openmetadata.slack.com/">Slack</a> if you have any questions about <a href="https://github.com/open-metadata/OpenMetadata">code</a>, <a href="https://docs.open-metadata.org/install/run-openmetadata">installation</a>, and <a href="https://docs.open-metadata.org/v/main/">docs</a>. For feature requests, please file a <a href="https://github.com/open-metadata/OpenMetadata/issues">GitHub issue</a> or contact us on <a href="https://openmetadata.slack.com/">Slack</a> for feature requests. Are you interested in contributing code? Here are some <a href="https://github.com/open-metadata/OpenMetadata/labels/good-first-issue">good starting issues</a> to get you going.
And don’t forget to support us with a <a href="https://github.com/open-metadata/OpenMetadata">GitHub star</a> if you like what we are doing! That will help OpenMetadata to reach a wider audience and to build a community to solve data problems together.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=97c1fb603bcf" width="1" height="1" alt=""><hr>
<a href="https://blog.open-metadata.org/openmetadata-1-1-0-release-97c1fb603bcf">OpenMetadata 1.1.0 Release</a> was originally published in <a href="https://blog.open-metadata.org/">OpenMetadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

OpenMetadata 1.1.0 Release

<h4>And unlock Collaboration, Data Quality, Data Insights, and more!</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wgESug4tSoxbnJH3Qc71uw.png"></figure><a href="https://www.amundsen.io/">Amundsen</a> is one of the OSS Data Catalogs that was developed by Lyft and was open-sourced in October 2019. It quickly became popular for solving data discovery and data governance challenges. However, in recent years, Amundsen’s development and growth have slowed down considerably. Without an active community and no clear <a href="https://www.amundsen.io/amundsen/roadmap/">roadmap</a> to address the emerging needs, the users of Amundsen are looking for alternatives in the OSS space.
OpenMetadata is redefining the modern metadata platform with a <a href="https://blog.open-metadata.org/announcing-openmetadata-20399b816e60">bold vision</a>. We have built a centralized metadata repository based on metadata specifications and APIs from the ground up. It is the foundation for innovation with several applications, such as Discovery, Collaboration, Governance, Data Quality, and Data Insights going beyond passive Data Catalogs. Learn more about <a href="https://open-metadata.org/">OpenMetadata’s</a> journey so far <a href="https://blog.open-metadata.org/openmetadata-release-1-0-the-journey-so-far-6b6d9319211e">here</a>.
<figure><img alt="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=small_rectangle" src="https://cdn-images-1.medium.com/max/700/0*LOsM5BY7xqy8bDJM.png"></figure>One of the frequent questions we get in our community is how can I migrate my existing <a href="https://www.amundsen.io/">Amundsen</a> installation to OpenMetadata without losing all the metadata built over the years. In this post, we will discuss the main differences between <a href="https://www.amundsen.io/">Amundsen</a> and <a href="https://open-metadata.org/">OpenMetadata</a>. We will showcase how easy it is to migrate all the knowledge stuck in <a href="https://www.amundsen.io/">Amundsen</a> to <a href="https://open-metadata.org/">OpenMetadata</a> and unlock many new capabilities.
<h3>Why should you Migrate?</h3>
While Amundsen is a Data Catalog, OpenMetadata is an <a href="https://docs.open-metadata.org/">Active Metadata Platform</a> with multiple applications built on top of it. Discovering the data is just the first step in a long chain of steps for any user:
<ul>
<li>Can I trust this data?</li>
<li>Does it fit my use case?</li>
<li>How are other people using it?</li>
<li>How can I request descriptions or tags?</li>
<li>How can I add a test to the tables I use to ensure data quality?</li>
</ul>
With the recent <a href="https://blog.open-metadata.org/openmetadata-release-1-0-the-journey-so-far-6b6d9319211e">1.0 Release</a>, <a href="https://open-metadata.org/">OpenMetadata</a> comes packed with core features around <a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-discovery">Discovery</a>, <a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-collaboration">Collaboration</a>, <a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-governance">Governance</a>, <a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-quality-profiler">Data Quality</a>, <a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-lineage">Lineage</a>, and <a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-insights">Data Insights</a>. Its goal? To be the single place to answer any questions users have about data and collaborate seamlessly.
Moving your data from Amundsen into OpenMetadata means improving your Data Discovery experience, unlocking new capabilities to profile your data, creating data quality tests and alerts, and setting up an environment to allow your organization to collaborate towards better data culture.
Moreover, the foundations of OpenMetadata revolve around the following:
<ol>
<li>
<a href="https://docs.open-metadata.org/v1.2.x/main-concepts/metadata-standard/schemas">Metadata Schema Specifications:</a> define a controlled vocabulary for metadata with types and constraints. Standardizing the language for metadata removes the need to reinvent the wheel when building metadata applications and opens the door for collaboration with other communities.</li>
<li>
<a href="https://docs.open-metadata.org/v1.2.x/main-concepts/metadata-standard/apis">Metadata API Specifications:</a> that power every aspect of the platform, from any click on the UI to the native Metadata Ingestion or Data Quality capabilities.</li>
<li>
Architectural Simplicity: built in the open-source space from the ground up, OpenMetadata requires only three dependencies, MySQL or Postgres for storing metadata, Elasticsearch or OpenSearch for powering search queries and schedulers like Airflow, Argo, Dagster, etc., to run the ingestion connectors. With fewer well-proven dependencies, it’s easy to deploy and manage production installations.</li>
<li>
<a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-collaboration">Collaboration</a>: Activity feeds to track all the changes on your metadata. Conversation threads to discuss these changes happening in your data. Request description/tags workflows. Create Tasks and assign them to owners of the data.</li>
<li>
<a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-quality-profiler">Native Data Quality:</a> Build trust in your data by creating tests with zero-code data quality in OpenMetadata. Learn more about <a href="https://youtu.be/gLdTOF81YpI">Native Data Quality in OpenMetadata</a>
</li>
<li>
<a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-insights">Data Insights: </a>Organizations can drive the adoption of OpenMetadata by monitoring its usage and setting up company-wide KPIs. The built-in goal-setting and tracking mechanisms help proactively drive your company’s data culture. You can define the Key Performance Indicators and set goals to be achieved within a timeframe towards better documentation, ownership, and tiering.</li>
<li>
Built on Modern Frameworks: OpenMetadata is built on top of modern frameworks, easy to operate on any public cloud or private cloud environment. You can deploy it on top of Kubernetes, Docker, or bare metal.</li>
<li>
Ingestion Framework: OpenMetadata has built <a href="https://docs.open-metadata.org/v1.0.0/connectors">62 connectors</a> to bring the metadata from various sources. Users can also build their own custom connectors easily. Here is a webinar showing how to <a href="https://youtu.be/fDUj30Ub9VE">build custom connectors</a>.</li>
<li>
Security and Compliance: As a Metadata platform, we know how important it is to keep the data secure. That’s why we invested in security from Day 1. OpenMetadata code goes through snyk.io, and GitHub Dependabot code scanning, and every release goes through container scanning. These reports are publicly available, allowing organizations to trust and deploy the software in production.</li>
<li>
<a href="https://docs.open-metadata.org/v1.2.x/deployment/security">SSO Support:</a> OpenMetadata provides support for all the major SSO integrations. Making it easy for anyone to integrate into their company’s SSO and bring their users and offer a secure solution.</li>
<li>
<a href="https://docs.open-metadata.org/v1.2.x/releases/roadmap">Development Velocity </a>— OpenMetadata is built on top of modern frameworks and has a release cycle of 4–6 weeks for the major version and every two weeks for the minor version. In the past 1.5 yrs, our community delivered 45 releases. In a short time, OpenMetadata has been adopted and deployed in 100s of companies and used by 1000s of users.</li>
<li>
Innovation — The OpenMetadata community has built many innovative features that no open-source and proprietary data catalogs can match. The Architecture and design decisions we made and our vibrant community will continue to innovate and set OpenMetadata apart from other platforms.</li>
</ol>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ufr94yATAlkRenVkB9AVoQ.png"><figcaption>Merged PRs by week for the last year between Amundsen &amp; OpenMetadata</figcaption></figure><h3>Come for the Vision and Stay for the Execution</h3>
OpenMetadata was <a href="https://blog.open-metadata.org/announcing-openmetadata-20399b816e60">announced on Aug 2021 with a grand vision</a>. We accomplished a large part of that vision with our <a href="https://blog.open-metadata.org/openmetadata-1-0-release-beb34762d916">1.0 Release</a> and continue to innovate and provide<a href="https://docs.open-metadata.org/v1.0.0/releases/roadmap"> a roadmap</a> of our goals by incorporating feedback and feature requests from the community.
By migrating to OpenMetadata, not only will you gain a platform that is easy to deploy, manage, secure, and packed with features, but you will also be a part of a fast-growing community executing a great vision and delivering new features with every release.
<h3>Migration Guide</h3>
Migrating any production service requires a lot of work and is often a cumbersome process. Our goal for the <a href="https://www.amundsen.io/">Amundsen</a> migration has been to make it as simple as possible and ensure that your teams’ work is not wasted. As our guide below will show, we migrate all the documentation, ownership, and tags into <a href="https://open-metadata.org/">OpenMetadata</a>. All your team’s work in Amundsen will be preserved and carried to OpenMetadata.
For this example, we will migrate some sample data from Hive, Delta, and DynamoDB.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iDSNeGamYKCl8m3VJkaq1Q.png"><figcaption>Amundsen metadata</figcaption></figure>If we look at one of the tables, we’ll see how it contains descriptions, owners, and tags:
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Iuhi4WBXozRJuzfsPhSYMA.png"><figcaption>Amundsen details view</figcaption></figure>Our goal is to ensure that all of this information is preserved once you move to <a href="https://open-metadata.org/">OpenMetadata</a>. The deployment process is out of scope for this guide, but you will find all the necessary information <a href="https://docs.open-metadata.org/v1.0.0/deployment">here</a>.
<h3>Step 1 — Create the Amundsen Service</h3>
A Service is an entity holding the connection information of a source we want to extract metadata. On the OpenMetadata UI, click on Settings; in the left panel, click on Metadata to add a new Service.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YAW9FazE20Lu7KkyhgGiCQ.png"><figcaption>Create the Amundsen service</figcaption></figure>In the connection information, add the credentials to connect to the underlying Neo4J instance.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CZ3dHx3bKXsnOchFO-R_Xg.png"><figcaption>Add the connection details</figcaption></figure>Once the Test Connection is successful, follow the steps to create and deploy the ingestion workflow. Since this will be a one-time action, you can schedule it to None, and we will trigger it manually in the next step.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6PXPkVW3Xm8KY_q2DubOzQ.png"><figcaption>Schedule the ingestion process</figcaption></figure><h3>Step 2 — Run the Metadata Ingestion</h3>
Navigating to the Service, you will see the control panel for your ingestion workflow. Click on the Run button and give it a few minutes until the run is Successful.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N5pkweVDi_uU8zfac0Vjxg.png"><figcaption>Run the Metadata ingestion</figcaption></figure><h3>Step 3 — Explore the Metadata</h3>
Once the run is complete, we will see three new Database Services in OpenMetadata: Hive, Delta, and DynamoDB.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GyATgJZCgGFQUdu3Ojexqw.png"><figcaption>Metadata in OpenMetadata</figcaption></figure>If we inspect the metadata we got from Hive, all the Amundsen tables will be inside, and we will be able to view the same information in the new platform.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f7IQnVVcX1W2LUUyLVYLTA.png"><figcaption>Hive details in OpenMetadata</figcaption></figure>For your convenience, all the tags migrated from Amundsen will be in the AmundsenTags Classification. You can then keep them there or explore the <a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-governance">Classification and Glossary capabilities</a> that OpenMetadata brings.
<h3>Step 4 — Updating the New Services</h3>
At this point, we have the metadata snapshot we brought from Amundsen. What happens if our sources are still being updated and we want to schedule metadata syncs? Let’s head to the UI to update the sources’ connections!
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OOMfaFugibFsT5btS9TIPw.png"><figcaption>Updating the Hive connection</figcaption></figure>For detailed documentation, please <a href="https://docs.open-metadata.org/v1.2.x/connectors/metadata/amundsen">refer here</a>
Click on Edit Connection to add the credentials to connect to each source and start scheduling the Metadata workflow. The good part is that for most of the Database Services, you can also create Profiler Workflows to gather further insights from your data and leverage the <a href="https://www.youtube.com/watch?v=gLdTOF81YpI&amp;t=2829s&amp;ab_channel=OpenMetadata">Data Quality features</a>.
<h3>Summary</h3>
OpenMetadata lays a solid foundation to fuel innovation in the metadata space. As a Metadata Platform, you can leverage <a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-discovery">Discovery</a>, <a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-collaboration">Collaboration</a>, <a href="https://docs.open-metadata.org/v1.2.x/how-to-guides/data-quality-profiler">Data Quality</a>, and many more applications in a single place, reducing metadata fragmentation and user frustration.
In this guide, we have learned how to migrate from <a href="https://amundsen.io/">Amundsen</a>, allowing us to use new and powerful tools while maintaining the context and knowledge built so far. The best part? It just takes a few minutes to set up.
If you want to learn more about OpenMetadata, visit our <a href="https://open-metadata.org/">site</a>! Don't hesitate to get in touch with us on <a href="https://slack.open-metadata.org/">Slack</a> with questions about code, installation, and docs. For feature requests, please file a <a href="https://github.com/open-metadata/OpenMetadata/issues/new/choose">GitHub issue</a> or contact us on <a href="http://openmetadata.slack.com/">Slack</a>. Are you interested in contributing code? Here are some <a href="https://github.com/open-metadata/OpenMetadata/labels/good-first-issue">good starting issues</a> to get you going.
<figure><img alt="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=large_rectangle" src="https://cdn-images-1.medium.com/max/700/0*OPQdSqn6DmUuKsyq.png"></figure><img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=6104cd2d5a71" width="1" height="1" alt=""><hr>
<a href="https://blog.open-metadata.org/stuck-with-amundsen-here-is-how-to-migrate-to-openmetadata-6104cd2d5a71">Stuck with Amundsen? Here is how to migrate to OpenMetadata</a> was originally published in <a href="https://blog.open-metadata.org/">OpenMetadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Stuck with Amundsen? Here is how to migrate to OpenMetadata

OpenMetadata Release 1.0 — The Journey So Far

<h4>Improved Schemas &amp; APIs, Ingestion Improvements, Storage Services, Dashboard Data Models, Auto PII Classification, Localization, and much more.</h4>
Written By: <a href="https://medium.com/u/16e0670af08f">Teddy Crépineau</a>, <a href="https://medium.com/u/e13d9840529d">Shailesh Parmar</a>, <a href="https://medium.com/u/ee50eb7d628e">Nahuel</a>, <a href="https://medium.com/u/9fe15fcc2248">Chirag M</a>, <a href="https://medium.com/u/7dc7409e63af">Suresh Srinivas</a>, <a href="https://medium.com/u/1869429441c7">Mayur Singal</a>, <a href="https://medium.com/u/5d3218cd196e">Pere Miquel Brull</a>, <a href="https://medium.com/u/ade15bdd6312">Mohit Yadav</a>, <a href="https://medium.com/u/806163d03dc1">Shilpa Vernekar</a>, <a href="https://medium.com/u/d366035928ab">Sriharsha Chintalapani</a>, <a href="https://medium.com/u/eebbacdc6b45">Karan Hotchandani</a>, <a href="https://medium.com/u/74fad1116d0a">Sachin chaurasiya</a> and OpenMetadata Community.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2n4pvHbUV4OJ78BpIONoqg.jpeg"></figure>The much-awaited OpenMetadata 1.0 Release is here. This has been a long exhilarating journey over 1.5 years, with 40 releases and 100s of features built with the help of 160 community developers and validation in 100s of installations across the world. With release 1.0, the majority of our vision is delivered — a best-in-class metadata platform that supports Discovery, Lineage, Collaboration, Governance, Data Quality, and Data Insights in a single platform.
The focus of this release has been to provide a stable and sturdy product, along with many improvements around metadata ingestion. A lot of work has gone into stabilizing and improving the Schemas and APIs, ensuring that the APIs are backward compatible. OpenMetadata’s ingestion framework has always supported a wide range of service connectors, and we are introducing a new one — <a href="https://docs.open-metadata.org/connectors/database/impala">Apache Impala. </a>Moreover, based on community feedback, the overall UI/UX has been improved for the ingestion and glossary.
The current release covers essential automation to tag PII and sensitive data to help companies comply with GDPR. Finally, we now support the concept of Data Models for Dashboard services to define and manage data within the dashboard tool itself.
<figure><a href="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=small_rectangle"><img alt="" src="https://cdn-images-1.medium.com/max/728/1*M_tTyWDVeLK72TX0_DklIg.png"></a></figure><h3>Community Update</h3>
A big THANK YOU to everyone involved in the OpenMetadata community, using the product and continuously giving us feedback, and helping us improve. We could not be happier with the growth and care of this community since we open-sourced OpenMetadata. All of this could not have been possible without you.
<ul>
<li>Crossed 2200+ GitHub stars
</li>
<li>The Slack community reached 2700+ members</li>
<li>
161 Open source GitHub developers
</li>
<li>
866 Commits were merged into the 1.0 Release
</li>
</ul>
<h3>OpenMetadata 1.0 Release Highlights</h3>
<h3>APIs &amp; Schema</h3>
In the 1.0 Release, we’ve stabilized and improved the Schemas and APIs. OpenMetadata has always been particular about the power of metadata APIs to transform the world of data. Well-designed APIs enable the reuse of the central standards-based metadata to simplify the existing tools currently burdened with maintaining their own duplicated fragmented copy of the metadata. Many applications and automation can be built using our APIs. With this release, we have stable and backward-compatible APIs for developers to innovate.
<h3>Ingestion</h3>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TdjLXJIZ4NWRXXtCmJAjmQ.gif"></figure>Connecting to your data sources has never been easier. You can directly find all the necessary permissions and connection details in the UI. The overall UX/UI has been improved to create new connections to your data sources. When integrating your systems, you have detailed contextual documentation within the app.
When testing the connection, every internal step of the metadata extraction process is tested to let you know which of the required permissions are missing. We now have a comprehensive list of validations to let you know which pieces of metadata can be extracted with the provided configuration. There’s a clear status displayed if all or only partial metadata will be ingested.
We have improved the performance when extracting metadata from sources such as Snowflake, Redshift, Postgres, and dbt. We are fetching as much bulk information as possible. We are providing more levers so you can tune how the ingestion behaves, allowing you to enable or disable the ingestion of tags or owners for any service.
We have improved the parsing process and the overall performance of dbt workflows. Support for the <a href="https://docs.open-metadata.org/connectors/database/impala">Apache Impala connector</a> is available. We will remove the Impala schemes from the Hive connector in the next release. OpenMetadata now supports secure LDAP connection without the MTLS (verification of server) requirement.
<h3>Storage Services</h3>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sFOjA68D0OLyvNVb7olhfQ.gif"></figure>Based on your <a href="https://github.com/open-metadata/OpenMetadata/discussions/8124">feedback</a>, we created a new service to extract metadata from your Cloud storage. The earlier Data Lake connector ingested one table per file, which covered only some of the use cases. <a href="https://youtu.be/DDjbUCjm00c">Storage Services</a> help us represent containers (e.g., Buckets) and their data structure, if any. The first implementation has been done on Amazon S3, wherein you can specify your tables and partitions and see them reflected with the rest of your metadata. This has been a major contribution from <a href="https://github.com/cristiancalugaru">Cristian Calugaru</a>, Principal Engineer at Forter. We will keep adding support for other storage sources in the upcoming releases.
<h3>Dashboard Data Models</h3>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2gBsdW8Mhze1FlVjFt2SAw.gif"></figure>Dashboard Services now support the concept of <a href="https://youtu.be/ous8gOfiAHE">Data Models</a>: data that can be directly defined and managed in the Dashboard tooling itself, e.g., LookML models in Looker. Data Models will help us close the gap between Engineering and Business by furnishing crucial metadata from sources typically used and managed by business users or analysts. The first implementation of data models has been done for Tableau and Looker.
<h3>Query as an Entity and UI Overhaul</h3>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pkMO80Lv3Af11QSTkaYZ6w.gif"></figure>The 1.0 Release has an Improved UI for <a href="https://youtu.be/tb5hEmzL1rY">SQL Queries</a>, with faster loading times and allowing users to vote for popular queries. Users can now create and share a Query directly from the UI, linking it to multiple tables if needed. Users can also discuss and react to the other queries in each table.
<h3>Localization</h3>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vqUfeXvdKDE2Klfic1To7w.gif"></figure>With more and more users from across the globe using OpenMetadata, we’ve added <a href="https://youtu.be/V9EaXEJ_Dxk">Localization Support</a> in the UI. Now you can use OpenMetadata in English, French, Chinese, Japanese, Portuguese, and Spanish. We would greatly appreciate <a href="https://docs.open-metadata.org/how-to-guides/how-to-add-language-support#how-to-add-language-support">community contributions</a> to the above-mentioned languages.
OpenMetadata uses ElasticSearch or OpenSearch to index all the metadata and provide the search options. Earlier, we only provided options to index and analyze documents in the English language. In this release, we’ve worked on a pluggable analyzer. The mapping is specific to each language and allows users to configure based on their language of choice.
<h3>Glossary</h3>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*g7NuETsKrtKdUkEMOwhHag.gif"></figure>The 1.0 release has a <a href="https://youtu.be/yeX3YY8IvLU">new and improved UI</a> for Glossary. <a href="https://youtu.be/mykm49U-QBk">Global search</a> has been enabled for Glossary terms and Tags, making it easier to discover data. Keeping in mind further changes and improvements that can take place in an organization, the Glossary Terms are no longer restricted to the original glossary they were created in. Users can now drag and drop Glossary Terms within and across Glossaries, as required. Instead of searching and tagging their assets individually, users can add Glossary Terms to multiple assets from the Glossary UI.
<h3>Auto PII Classification</h3>
The European Union’s data protection law <a href="https://gdpr-info.eu/">General Data Protection Regulation</a> (GDPR) requires companies to follow strict rules for data protection, especially around personal data. Keeping the GDPR requirements in mind, OpenMetadata has implemented an automatic way to tag PII and sensitive data. The auto-classification is an optional step of the Profiler workflow. We will analyze the column names, and if sample data is being ingested, we will run NLP models on top of it.
<h3>Search</h3>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vw5Kc5WvWC-IOah8FN8xwQ.gif"></figure>Several improvements have been made to data discovery. We’ve Improved Relevancy with added support for partial matches. The ranking has been improved with the most used or higher Tier assets at the top of the search. Support has been added for Classifications and Glossaries in the global search.
<h3>Security</h3>
In this release, SAML support has been added.
Apart from that, we have an important Deprecation Notice: The SSO Service accounts for Bots will be deprecated. Going forward, JWT authentication will be the preferred method for creating Bots.
<h3>Lineage</h3>
The Lineage UI has been enhanced to display a large number of nodes. Users can view nearly 1000+ nodes. The UI has been improved for better navigation. Also, improvements have been made to the SQL parser to extract lineage in the Lineage Workflows.
<h3>Chrome Browser Extension</h3>
All the metadata is at your fingertips while browsing Looker, Superset, etc., with the OpenMetadata Chrome Browser Extension. The Chrome extension supports Google SSO, Azure SSO, Okta, and AWS Cognito authentication. You can Install the <a href="https://chrome.google.com/webstore/detail/openmetadata-chrome-exten/pakbbdhbbiclnceabdmnghamabjloofc">Chrome extension</a> from the Chrome Web Store.
<h3>Other Changes</h3>
<ul>
<li>We now support long entity names (E.g., the S3 paths)</li>
<li>Support has been added to import as well as export all the entities, along with the ability to propagate tags during the process.</li>
<li>The Explore page cards will now display a maximum of ten tags. Users will still be able to view the other tags if needed.</li>
<li>Entity names support apostrophes.</li>
<li>Improvements have been made to the Summary panel to be consistent across the UI.</li>
</ul>
<h3>Planned for 1.1 Release</h3>
For the latest updates on our next release, please refer to the <a href="https://docs.open-metadata.org/overview/roadmap">Roadmap</a>.
Entities
<ul>
<li>Support for NoSQL, MongoDB, ElasticSearch connectors</li>
<li>Support for registering Services/Applications along with OpenAPI schema</li>
</ul>
Landing Page — Customizable Landing page for different personas of the Users.
Bulk Functionality
<ul><li>Upload descriptions, and tags to all the entities via import/export functionality</li></ul>
Data Quality
<ul>
<li>Quality Page UI improvements</li>
<li>Test failure propagation and view quality in the lineage UI</li>
</ul>
<h3>Thanks to our Contributors</h3>
A huge shoutout to our contributors for their code contributions and super helpful feedback from the start of this open-source project. Your active participation has made a major difference to the progress of OpenMetadata. Thank you for all that you have done.
Thank you <a href="https://github.com/cristiancalugaru">Cristian Calguru</a> for building the much-awaited Storage Service and S3 connector. Thank you <a href="https://github.com/mosiac1">Cristian Osiac</a>, <a href="https://github.com/ksirmons">Keith Sirmons</a>, <a href="https://github.com/Hemal-Mamtora">Hemal Mamtora</a>, <a href="https://github.com/bohanhou14">Bohan Hou</a>, <a href="https://github.com/heroWang">Hawkins</a>, <a href="https://github.com/BirdboyBolu">Boluwatife Victor</a>, <a href="https://github.com/pdeeparao">Deepa Rao</a>, <a href="https://github.com/osocron">Noe Alejandro Perez Dominguez</a>, <a href="https://github.com/geoHeil">Georg Heiler</a>, <a href="https://github.com/krotonet">Krotonet</a>, <a href="https://github.com/rglbr">Ragul Balaji Ravichandran</a>, <a href="https://github.com/joeshajan">Joe Shajan</a>, <a href="https://github.com/yu-iskw">Yu Ishikawa</a>, <a href="https://github.com/JINZHANG2017">JINZHANG2017</a>, <a href="https://github.com/stephanesol">Stéphane Sol</a>, <a href="https://github.com/VolkovGeoPhy">VolkovGeoPhy</a>, <a href="https://github.com/awittzb">Austin Witt</a>, <a href="https://github.com/nuang-ee">Sasha Chung</a>, <a href="https://github.com/kcd83">Kcd83</a>, <a href="https://github.com/SuperNascher">Kevin R</a>, <a href="https://github.com/charliemenke">Charlie Menke</a>, <a href="https://github.com/schlameel">John McCormick</a>, <a href="https://github.com/megumiie">Megumi AIKAWA</a>, <a href="https://github.com/jan-pieter">Jan-Pieter van den Heuvel</a>, <a href="https://github.com/Anuj359">Anuj359</a>, and <a href="https://github.com/luis-fnogueira">Luis Felipe Almeida Nogueira</a>.
Thanks to <a href="https://github.com/allenhaozi">Allen Haozi</a>, <a href="https://github.com/mosiac1">Cristian Osiac</a>, <a href="https://github.com/cvlendistry">cvlendistry</a>, <a href="https://github.com/EminUZUN">Emin Uzun</a>, <a href="https://github.com/ehausig">Eric Hausig</a>,
<a href="https://github.com/Beetelbrox">Francisco J. Jurado Moreno</a>, <a href="https://github.com/geoHeil">Georg Heiler</a>, <a href="https://github.com/HaithemSouala">Haithem Souala,</a> <a href="https://github.com/knoerregaard">Klaus Noerregaard</a>, <a href="https://github.com/lailapatel">Laila Patel</a>, <a href="https://github.com/trillhaa">Martin Trillhaas</a>, <a href="https://github.com/mukheshnarra">Mukhesh Narra</a>, <a href="https://github.com/najdalq">Najd Alqahtani</a>, <a href="https://github.com/rogerzs">Rogério Ferreira Dos Santos</a>, and <a href="https://github.com/sivaarumalla">Siva Arumalla</a> for raising GitHub issues that made it to the 1.0 release. Thank you <a href="https://github.com/abcabhishek">abcabhishek</a>, <a href="https://github.com/chethan-bs">Chethan B S</a>, and <a href="https://github.com/vsapucaia">Vitor Fortunato</a> for your feedback on the GitHub issues included in this release.
Please reach out to us on <a href="https://openmetadata.slack.com/">Slack</a> if you have any questions about <a href="https://github.com/open-metadata/OpenMetadata">code</a>, <a href="https://docs.open-metadata.org/install/run-openmetadata">installation</a>, and <a href="https://docs.open-metadata.org/v/main/">docs</a>. For feature requests, please file a <a href="https://github.com/open-metadata/OpenMetadata/issues">GitHub issue</a> or reach out to us on <a href="https://openmetadata.slack.com/">Slack</a>. Interested in contributing code? Here are some <a href="https://github.com/open-metadata/OpenMetadata/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">good starting issues</a> to get you going.
Please give us a <a href="https://github.com/open-metadata/OpenMetadata">GitHub star</a> if you like what we are doing. That will help OpenMetadata to reach a wider audience and to build a community to collaboratively solve data problems.
<figure><a href="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=large_rectangle"><img alt="" src="https://cdn-images-1.medium.com/max/728/1*B5kZqafplTME5JQ_COWGwA.png"></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=beb34762d916" width="1" height="1" alt=""><hr>
<a href="https://blog.open-metadata.org/openmetadata-1-0-release-beb34762d916">OpenMetadata 1.0 Release</a> was originally published in <a href="https://blog.open-metadata.org/">OpenMetadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

OpenMetadata 1.0 Release

<h4>Extracting Metadata from Every Corner: The evolution of the Ingestion Framework at OpenMetadata</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*03tStYE0nxgBn80o"><figcaption>How we built the Ingestion Framework</figcaption></figure>The first step for any user of OpenMetadata is to connect to their sources. We are talking about an ever-growing list of currently 60 services that we need to extract metadata from, translate it to the OpenMetadata standard, and send it to the server.
<figure><a href="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=small_rectangle"><img alt="" src="https://cdn-images-1.medium.com/max/728/1*M_tTyWDVeLK72TX0_DklIg.png"></a></figure>Without metadata, there are no discovery, collaboration, or quality tests. The ingestion process is a requirement that unlocks the rest of the features, and we are constantly pushing for improvements.
In the last year, two major transformations have helped us bring the Ingestion Framework to the next level. This post will describe the tools, decisions, and designs that made the Ingestion Framework simpler, more robust, and easier to evolve.
<h3>High-Level Design</h3>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jOhRdp33aSR8LDaJ"><figcaption>High-level design: Source, Process, and Sink</figcaption></figure><blockquote>At OpenMetadata, every action is an API call, and the Ingestion Framework is no different.</blockquote>
A Python package defines the process that loads metadata into the OpenMetadata server. Of course, many pieces and utilities are involved, such as the OpenMetadata client — a typed wrapper of the server APIs. Still, if we stick to the core ideas, we need to answer a few questions:
<ol>
<li>How do we interact with all the different sources?</li>
<li>How do we generalize and centralize the process and communications with the server?</li>
<li>How do we allow this process to run anywhere?</li>
</ol>
The first question is essential but does not impact the design nearly as much as the other two: We need a generic module to help us grow the list of supported sources while not tying us to specific orchestration tools.
<h3>Separation of Concerns</h3>
We achieved a method that’s generic enough by defining a Workflow class as a step-wise process that simplifies and moves data from a Source into a Sink. This allows us to put the scope on pieces that are independent of each other.
Thanks to the OpenMetadata standard, each Source only needs to focus on how to extract and translate metadata but does not really care about what happens next. In the same line of thought, the Sink class will only handle how the metadata is sent to the OpenMetadata server.
The Workflow class can then be created dynamically based on a JSON structure specifying the Source, Sink, and how to connect to the OpenMetadata Server.
<pre>{ "source": { … }, "sink": { … }, "workflowConfig": { … } }</pre>
Therefore, any orchestrator with a Python environment can load such a JSON and run the workflow definitions from the openmetadata-ingestion <a href="https://pypi.org/project/openmetadata-ingestion/">package</a>.
<h3>Simple is Better than Complex</h3>
After we defined an ingestion structure that worked, the main goal was to put it out in the open. We needed our community to have a fast and easy way to ingest metadata so they could start exploring the rest of the platform’s benefits. Thus, the ingestion team created a CLI that ran the workflow from a JSON file like the above.
However, it was not sustainable. For a few releases, the number of issues and questions related to the ingestion was never-ending. We spilled internal logic for the users to understand and relied on their laptops or host environments to run the process. We were paying the price of forcing a complex step before users could use the rest of the tool.
Democratizing (meta)data means building products that aren’t only useful for developers: our next goal was making the ingestion process available from the UI. This would give us control over the end-to-end flow and let our users focus on what they want, not on what they need to get there.
<h3>Containers, Connections, and Dynamic DAGs</h3>
The idea was simple: the server should be the only point of contact for every interaction, including the metadata workflows. Any user should be abstracted entirely over the underlying tooling and have complete control in the UI to schedule new ingestions, check the logs, or rerun them manually.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1Ttxmfh8RFbXl89g"><figcaption>Managing Ingestion Pipelines in the OpenMetadata UI</figcaption></figure>The tool of choice for the OSS version has been Airflow, which is widely known and has already been adopted in several organizations. However, the exciting challenge here was that Airflow did not provide the main functionality we needed: deploying DAGs dynamically from an API call. So we built our own custom <a href="https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-airflow-apis">plugin</a>, which is now the communication channel between Airflow and the OpenMetadata server.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l79CpSow2ug5Cdss"><figcaption>UI, Server, and Workflow interactions</figcaption></figure>Thanks to the Airflow <a href="https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/plugins.html">plugins</a>, we could define a set of APIs on top of the Ingestion Framework: from deploying a DAG to testing the connection to a service. This meant we now needed to define the requirements to connect to each service globally. The Ingestion Framework was not the only one that needed to be aware of the parameters to reach a database or connect to an external API. These connections were now filled in the UI and sent over to Airflow to be run.
As we were already using JSON Schema for the OpenMetadata standard, we migrated all the connection definitions into the <a href="https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections">specification</a>, making them available to every part of the code. This broader use of service connections also unlocked another element to be tackled: Security. This is a topic extensive enough to require an entire post, so for now, we’re going to link to the <a href="https://docs.open-metadata.org/deployment/secrets-manager">docs</a> for curious minds.
<h3>Service Topologies</h3>
The last piece of the puzzle was not related to the UX or architecture but to the nature of the problem at hand: How could we safely maintain and evolve a list of sources that will only grow? Moreover, as an open-source project, how can we make contributors’ lives easier?
We needed an approach checking the following boxes:
<ol>
<li>It should be flexible enough to allow us to perform generic improvements and iterations to the whole framework without many reworks.</li>
<li>We should standardize the requirements for each service type. We don’t have the same metadata when ingesting a database source vs. an ML Model.</li>
<li>Developers should have precise recipes to follow to add new sources.</li>
</ol>
Abstract classes help the interpreter notice if a specific method is missing in an implementation. Still, the extraction process has a particular order (hierarchy) that must also be followed. For example, we can only ingest a table once its schema has been processed, and the lineage of a dashboard requires the dashboard itself to be present in OpenMetadata first.
Each service must follow a series of extraction steps to be executed in a specific order. This idea was the foundation of Service Topologies: a sequence of Nodes and Stages that define in which order the metadata is extracted from the Source and which transformations are applied to produce the OpenMetadata assets.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NhCi3dcu0ICp8AHH"><figcaption>Pipeline Service Topology</figcaption></figure>If we look at the <a href="https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/pipeline/pipeline_service.py#L51">Pipeline Topology</a>, we will first get and create the Pipeline Service. Afterward, we will obtain the list of pipelines to be ingested and iterate over them to create the pipeline assets in OpenMetadata and send their tags, status, or lineage information.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K7QODVGyGzCb3X2B"><figcaption>Database Service Topology</figcaption></figure>The same thing happens when talking about <a href="https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/database_service.py#L95">Database Services</a>. The nodes guide us through the asset hierarchy: Service -&gt; Database -&gt; Schema -&gt; Table. Order matters, as we can only create a table once we know the schema it belongs to. These ingredients are then shared in the execution context: the Topology also helps us define the required information for processing each Stage.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uEGY2uHUjeTKPOLO"><figcaption>High-level Topology description</figcaption></figure>Python dynamism played a crucial role in this solution, as we can pick up <a href="https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/api/topology_runner.py#L68">on-the-fly</a> the methods that need to be executed at each moment. Updating the service extraction becomes as easy as adding a new abstract method to be implemented and its name in the Topology as a string at the exact position that the execution needs to happen.
<h3>Summary</h3>
In this post, we’ve had a glimpse of the evolution of the Ingestion Framework during this last year. Our major focus has been improving the UX with the UI integration of the ingestion while also focusing on the internal design to make the code easier to maintain and evolve.
Code is never finished, only <a href="https://en.wikiquote.org/wiki/Paul_Val%C3%A9ry#Charmes_ou_po%C3%A8mes_%281922%29:~:text=A%20poem%20is%20never%20finished%3B%20it%20is%20only%20abandoned.">abandoned</a>, and we will keep working to make the Ingestion Framework better release after release. These improvements have been essential, but there is still room for improvements to make the ingestion faster and more robust.
If you find this interesting, want to chip in, or just say hi, do not hesitate to reach out! Please take a look at our GitHub <a href="https://github.com/open-metadata/OpenMetadata">repo</a>, give it a star, and drop us a line in <a href="https://slack.open-metadata.org/">Slack</a> 🚀
<figure><a href="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=large_rectangle"><img alt="" src="https://cdn-images-1.medium.com/max/728/1*B5kZqafplTME5JQ_COWGwA.png"></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=1af0b6ff5c81" width="1" height="1" alt=""><hr>
<a href="https://blog.open-metadata.org/how-we-built-the-ingestion-framework-1af0b6ff5c81">How we Built the Ingestion Framework</a> was originally published in <a href="https://blog.open-metadata.org/">OpenMetadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

How we Built the Ingestion Framework

<h4>Improved SQL Lineage, Glossary Bulk Upload, Unified Tag Category API, Mutually Exclusive Tags, Chrome Extension, and lots more.</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N9OjxMlZZBRwxgR3kCLq4Q.png"></figure>We are glad to announce OpenMetadata’s Release — 0.13.2, which showcases some exciting features and improvements like Glossary bulk upload, improved SQL lineage, mutually exclusive tags, and a beta release of the OpenMetadata Chrome extension.
The OpenMetadata community recently hosted a <a href="https://www.youtube.com/watch?v=fDUj30Ub9VE&amp;t=9s">Mini-Webinar on Custom Connectors</a>, which discussed the core aspects of OpenMetadata, the ingestion process, and building your own custom connectors. Most organizations have their own in-house systems and other third-party services crucial to their business. OpenMetadata supports 60+ connectors, and by building a custom connector, you can have a holistic view of everything that happens in your Data Platform.
We plan to add additional features to the <a href="https://www.getcollate.io/">SaaS</a> version in future releases, so stay tuned for more updates 🚀.
<figure><a href="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=small_rectangle"><img alt="" src="https://cdn-images-1.medium.com/max/728/1*M_tTyWDVeLK72TX0_DklIg.png"></a></figure><h3>Community Update</h3>
Thanks to OpenMetadata’s amazing community. We’ve been receiving productive feedback and active code contributions over the past one and a half years. And, we deeply appreciate your support and feedback.
<ul>
<li>Crossed 1800+ GitHub stars</li>
<li>The Slack community reached 2200+ members</li>
<li>141 Open source GitHub developers</li>
<li>312 Commits were merged into the 0.13.2 release</li>
</ul>
<h3>OpenMetadata 0.13.2 Release Highlights</h3>
<h4>Glossary Import and Export</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9cBtzHcH-URSL8hVdcwzRg.gif"><figcaption>Glossary import and export</figcaption></figure>When the Glossary was introduced in the 0.9 release, we supported a manual approach to add the glossary terms one at a time. This can become time-consuming for organizations already maintaining a business glossary in a spreadsheet or similar tools. Based on community feedback, we’ve introduced the functionality to bulk upload glossary terms. Users can save time and effort by uploading a CSV with thousands of terms in one go. The import utility will validate the file and show you a preview of the elements that will be imported to OpenMetadata. Just as you can import the glossary terms, you can export the glossary data as a CSV file.
We take your feedback really seriously, and we are always excited to see community requests come to life!
<h4>Improved SQL Lineage</h4>
Open source thrives when communities grow together. We have been heavy users of <a href="https://github.com/reata/sqllineage">sqllineage</a> for our query parsing processes, and it’s been a great tool. Still, the growing number of connectors and community needs required us to improve our existing parsing capabilities.
We have worked together with <a href="https://github.com/reata">Reata</a>, the developer behind sqllineage, to migrate the library’s core to <a href="https://www.sqlfluff.com/">sqlfluff</a>. We’d like to take this opportunity to thank both communities for merging our contributions and helping bring our lineage parsing to the next level.
If you are interested in learning more about how we handle Data Lineage in OpenMetadata, you can look at our <a href="https://www.youtube.com/watch?v=jEbN1tt89H0&amp;t=1s&amp;ab_channel=OpenMetadata">webinar</a>!
<h4>New Glossary UI</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MutjL9RNYkZrF2jR"></figure>Previously, the Glossary and the Glossary terms were displayed together in a tree UI. Now, we’ve separated the Glossary and Glossary Terms within the UI, such that the Glossaries are displayed in the left navigation pane, while the glossary details are displayed on the right. The list of terms has been sorted alphabetically for better navigation. The improved UI changes will help you easily access the details with tags and descriptions in the cards.
<h4>From Tag Category to Classification</h4>
We have renamed Tag Categories to Classification as a more widely used term for expressing groups of tags. The team also took this chance to unify the Tags API to conform to the other APIs we have been building.
This change will make it easier for developers familiar with the existing APIs, and it simplified different internal flows. You can find more information about the exact modifications <a href="https://github.com/open-metadata/OpenMetadata/issues/9259">here</a>.
<h4>Mutually Exclusive Tags from the UI</h4>
There are situations where mixing multiple tags from the same classification does not make sense. Imagine an asset being tagged both with PII Sensitive and PII Non-Sensitive!
When creating a Classification or a Glossary term, their elements can be mutually exclusive. If this configuration is enabled, you won’t be able to assign multiple tags from the same category or glossary to the same data asset.
<h4>Chrome Extension</h4>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XBNCJuh-SXV1WtkVLKd0QQ.gif"><figcaption>OpenMetadata Chrome Extension</figcaption></figure>We are elated to announce the launch of the BETA version of the <a href="https://chrome.google.com/webstore/detail/openmetadata-chrome-exten/pakbbdhbbiclnceabdmnghamabjloofc?hl=en&amp;authuser=0">OpenMetadata Chrome extension</a>. Feel free to explore it and share your feedback. We are focused on meeting the needs of the Data Community, by simplifying processes and making all the metadata ubiquitously accessible by providing the necessary context in all your tools.
<h4>Connectors</h4>
<ul>
<li>
Databricks Pipeline Connector has been added as a community contribution!</li>
<li>Some important fixes have been made for the AWS QuickSight connector.</li>
<li>Improvements have been made to DB2 constraints and profiler.</li>
<li>The Oracle connector now ships with the Thick mode enabled.</li>
<li>Added support for Postgres Foreign Tables</li>
<li>Connectors like Redshift or Snowflake are way faster now, as we get the descriptions in batches.</li>
<li>Added support for Data Lake profiler row-based sampling.</li>
</ul>
<h4>Entity Names</h4>
The team introduced an enforcement of the Entity Names’ format using regex patterns. The intention is to better manage and harmonize Entity Names values and to allow users to form better expectations around them.
You can find more information about this change in the <a href="https://docs.open-metadata.org/deployment/upgrade">docs</a>.
<h3>Planned for Next Release</h3>
Our next release will be the long-awaited 1.0 Release ✨
The team has already started developing amazing new features, such as:
<ul>
<li>
Storage Service revamp to better represent your Datalakes (based on your <a href="https://github.com/open-metadata/OpenMetadata/discussions/8124">feedback</a>) with awesome community members!</li>
<li>NoSQL stores API definitions to prepare the ingestion for sources like Elasticsearch,</li>
<li>Automatic PII tagging during the profiler workflow,</li>
<li>Continuous improvement for the brand-new sqllineage package</li>
</ul>
And more! Stay tuned for any news, and let us know which features you’d like to see next!
<h3>Thanks to our Contributors</h3>
It’s been great to associate with the open-source community members for the last one and a half years. We’ve been receiving encouraging feedback, and great code contributions; all this has been a huge motivation to keep scaling new heights. Thanks to <a href="https://github.com/Aashish1221">Aashish1221</a>, <a href="https://github.com/Alina-Valea-Forter">Alina Valea Forter</a>, <a href="https://github.com/bigdata-spec">Bigdata-spec</a>, <a href="https://github.com/fredericocotrim">Frederico Cotrim</a>, <a href="https://github.com/geoHeil">geoHeil</a>, <a href="https://github.com/gnomolio">Gnomolio</a>, <a href="https://github.com/itaise">Itai Sevitt</a>, <a href="https://github.com/lailapatel">Laila Patel</a>, <a href="https://github.com/trillhaa">Martin Trillhaas</a>, <a href="https://github.com/NithinKumaraNT">Nithin Kumara N T</a>, <a href="https://github.com/pauloventurab">Paulo Ventura</a>, <a href="https://github.com/sergiopassoscb">Sérgio Passos</a> for raising GitHub issues that made it to the 0.13.2 release.
Please contact us on <a href="https://openmetadata.slack.com/">Slack</a> if you have questions about <a href="https://github.com/open-metadata/OpenMetadata">code</a>, <a href="https://docs.open-metadata.org/install/run-openmetadata">installation</a>, and <a href="https://docs.open-metadata.org/v/main/">docs</a>. For feature requests, you can file a <a href="https://github.com/open-metadata/OpenMetadata/issues">GitHub issue</a> or reach out to us on <a href="https://openmetadata.slack.com/">Slack</a>. Interested in contributing code? Here are some <a href="https://github.com/open-metadata/OpenMetadata/issues?q=is%3Aopen+is%3Aissue+label%3Agood-first-issue">good first issues</a> to get you going.
Please give us a GitHub star if you like what we are doing. That’ll greatly help OpenMetadata in reaching a wider audience.
<figure><a href="https://www.getcollate.io/?pk_campaign=14_day_trial&amp;pk_source=medium&amp;pk_medium=large_rectangle"><img alt="" src="https://cdn-images-1.medium.com/max/728/1*B5kZqafplTME5JQ_COWGwA.png"></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=e32c0de93361" width="1" height="1" alt=""><hr>
<a href="https://blog.open-metadata.org/openmetadata-0-13-2-release-e32c0de93361">OpenMetadata 0.13.2 Release</a> was originally published in <a href="https://blog.open-metadata.org/">OpenMetadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

OUR BLOG

Explore our Journey into the Data Collaboration Landscape