The Down-Low on Data Sources

<p>Data sources need love also! Trying to find and time the correct data source automation pipeline to your initiative can be a challenge. </p>

<div data-elementor-type="wp-post" data-elementor-id="1492" class="elementor elementor-1492" data-elementor-post-type="blog"> <section class="elementor-section elementor-top-section elementor-element elementor-element-cefaf24 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="cefaf24" data-element_type="section"> <div class="elementor-container elementor-column-gap-default"> <div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-60b30b1" data-id="60b30b1" data-element_type="column"> <div class="elementor-widget-wrap elementor-element-populated"> <div class="elementor-element elementor-element-828cf98 elementor-widget elementor-widget-text-editor" data-id="828cf98" data-element_type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p><span style="font-weight: 400;">A brief explanation of data sources and answers to some common questions our clients typically have about them.</span></p> </div> </div> <div class="elementor-element elementor-element-2064698 elementor-widget elementor-widget-image" data-id="2064698" data-element_type="widget" data-widget_type="image.default"> <div class="elementor-widget-container"> <img decoding="async" src="https://cdn.datalakehouse.org/media/sites/3/Blog-Post-Image-Data-Sources.png" title="" alt="" loading="lazy" /> </div> </div> <div class="elementor-element elementor-element-9c97572 elementor-widget elementor-widget-spacer" data-id="9c97572" data-element_type="widget" data-widget_type="spacer.default"> <div class="elementor-widget-container"> <div class="elementor-spacer"> <div class="elementor-spacer-inner"></div> </div> </div> </div> <div class="elementor-element elementor-element-42cf862 elementor-widget elementor-widget-text-editor" data-id="42cf862" data-element_type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p><span style="font-weight: 400;">No matter what industry you&#8217;re in, the chances are that you&#8217;re probably generating data (from multiple different sources) all over the place. And that can be a major pain to deal with when you&#8217;re trying to gather all the information you need in order to determine what’s working and what’s not, or simply present results to the necessary stakeholders. Combining all of the different data sources you&#8217;re using into a central location is one of the biggest challenges most businesses face today. </span></p><p><span style="font-weight: 400;">Your data sources can be on-prem or in the cloud and may include some combination of a finance and accounting system (</span><a href="https://quickbooks.intuit.com/accounting/" target="_blank" rel="noopener"><span style="font-weight: 400;">QuickBooks</span></a><span style="font-weight: 400;">, </span><a href="https://www.netsuite.com/portal/products/erp/financial-management/finance-accounting.shtml" target="_blank" rel="noopener"><span style="font-weight: 400;">NetSuite</span></a><span style="font-weight: 400;">, etc.), a CRM system (</span><a href="https://www.salesforce.com/" target="_blank" rel="noopener"><span style="font-weight: 400;">Salesforce</span></a><span style="font-weight: 400;">, </span><a href="https://www.hubspot.com/products/crm#" target="_blank" rel="noopener"><span style="font-weight: 400;">HubSpot</span></a><span style="font-weight: 400;">, etc.), a manufacturing system (</span><a href="https://katanamrp.com/" target="_blank" rel="noopener"><span style="font-weight: 400;">Katana</span></a><span style="font-weight: 400;">, </span><a href="https://www.aiscorp.com/" target="_blank" rel="noopener"><span style="font-weight: 400;">FactoryLogix</span></a><span style="font-weight: 400;">, etc.), a POS system (</span><a href="https://squareup.com/us/en" target="_blank" rel="noopener"><span style="font-weight: 400;">Square</span></a><span style="font-weight: 400;">, </span><a href="https://www.heartland.us/products/point-of-sale/restaurant" target="_blank" rel="noopener"><span style="font-weight: 400;">Heartland</span></a><span style="font-weight: 400;">, etc.), an inventory management system (</span><a href="https://www.fishbowlinventory.com/small-business-inventory-software" target="_blank" rel="noopener"><span style="font-weight: 400;">Fishbowl</span></a><span style="font-weight: 400;">, </span><a href="https://www.orderhive.com/" target="_blank" rel="noopener"><span style="font-weight: 400;">Orderhive</span></a><span style="font-weight: 400;">, etc.), excel spreadsheets, and/or a variety of other types of software. These are all sources full of data, with your end goal being to bring all of that juicy data together in order to turn it into useful information, information that will ultimately guide you to make the best business decisions possible. </span></p> </div> </div> <div class="elementor-element elementor-element-d232681 elementor-widget elementor-widget-image" data-id="d232681" data-element_type="widget" data-widget_type="image.default"> <div class="elementor-widget-container"> <a href="https://unsplash.com/photos/hpjSkU2UYSU" target="_blank" rel="noopener"> <img decoding="async" src="https://cdn.datalakehouse.org/media/sites/3/carlos-muza-hpjSkU2UYSU-unsplash.jpg" title="" alt="" loading="lazy" /> </a> </div> </div> <div class="elementor-element elementor-element-f8087b9 elementor-widget elementor-widget-spacer" data-id="f8087b9" data-element_type="widget" data-widget_type="spacer.default"> <div class="elementor-widget-container"> <div class="elementor-spacer"> <div class="elementor-spacer-inner"></div> </div> </div> </div> <div class="elementor-element elementor-element-9d1b739 elementor-widget elementor-widget-heading" data-id="9d1b739" data-element_type="widget" data-widget_type="heading.default"> <div class="elementor-widget-container"> <h2 class="elementor-heading-title elementor-size-default">Common Questions </h2> </div> </div> <div class="elementor-element elementor-element-d978557 elementor-widget elementor-widget-text-editor" data-id="d978557" data-element_type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p><span style="font-weight: 400;">We’re typically asked the same 5 questions about data sources and how they can be used: </span></p><ul><li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Can I get access to the data? </span></li></ul><p><span style="font-weight: 400;">You can, it depends on your source system. What you&#8217;re looking for is an access point. This could be as simple as downloading a CSV file onto a shared drive or data lake storage. Or, it could involve programmatic access to that data source system’s application programming interface (API).</span></p><ul><li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Is the data of good quality? </span></li></ul><p><span style="font-weight: 400;">Data Quality (DQ, and no I’m not talking about the ice cream 🍦😂) is an extremely important concept in data management. If the data is not in the correct format, or is missing data or values for certain rows or columns, for example, then turning the data into robust usable information may involve going through certain cleansing or transformation steps, before it can be reliably used.</span></p><ul><li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">What’s the best way to give everyone access to the data? </span></li></ul><p><span style="font-weight: 400;">This can be a tricky question. It depends on your information technology architecture. Or, if that’s too slow, nonexistent, or full of red tape, you’ll need a dependable self-service approach (especially when talking about data sources that are in the cloud). Other solutions like </span><a href="https://www.dropbox.com/?_hp=c&amp;landing=dbv2" target="_blank" rel="noopener"><span style="font-weight: 400;">Dropbox</span></a><span style="font-weight: 400;">, </span><a href="https://www.box.com/" target="_blank" rel="noopener"><span style="font-weight: 400;">Box</span></a><span style="font-weight: 400;">, or even a shared drive on your network can help. Worst case scenario, there’s always email, but I think you can imagine the inherent problems that go along with that 😏  </span></p><ul><li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Who owns the data?</span></li></ul><p><span style="font-weight: 400;">It depends, the data can be at a departmental level, enterprise level (meaning the entire organization), or at an individual level (think of a business analyst). Not all data is meant for all audiences, but the originating source of the data should always have a responsible person who’s a subject matter expert on that data. Other systems and means for delivering the data, transforming the data, and reporting on the data can allow the owner to secure and distribute the data as needed.   </span></p><ul><li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">What is data granularity?</span></li></ul><p><span style="font-weight: 400;">This is the level at which the data source exists. For example, data can be created at the day, week, month, quarter, year, etc. level, depending on the frequency of the demand for the data. This could be the case for most transactional data, like accounting and finance, but other transactions or event-based data (like internet of things (IoT) data) can get really deep with the frequency of events/transactions coming in at the second and even nanosecond levels. </span></p> </div> </div> <div class="elementor-element elementor-element-b7ed804 elementor-widget-divider--view-line elementor-widget elementor-widget-divider" data-id="b7ed804" data-element_type="widget" data-widget_type="divider.default"> <div class="elementor-widget-container"> <div class="elementor-divider"> <span class="elementor-divider-separator"> </span> </div> </div> </div> <div class="elementor-element elementor-element-777eaa6 elementor-widget elementor-widget-text-editor" data-id="777eaa6" data-element_type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p><span style="font-weight: 400;">If you thought all data sources were all just excel files and CSVs (although these are very common data sources and structures), you were sorely mistaken. There are lots of more advanced data source types/structures like JSON, SQL/Databases, Parquet, AVRO (don&#8217;t worry, we&#8217;ll go into these later in a more technical article).</span></p> </div> </div> <div class="elementor-element elementor-element-894e816 elementor-widget elementor-widget-image" data-id="894e816" data-element_type="widget" data-widget_type="image.default"> <div class="elementor-widget-container"> <a href="https://www.datalakehouse.io/"> <img loading="lazy" decoding="async" width="73" height="74" src="https://public-cdn.dlh.io/migrated/uploads/2021/06/brandonchan.png" class="attachment-thumbnail size-thumbnail wp-image-1434" alt="brandonchan" /> </a> </div> </div> <div class="elementor-element elementor-element-9be7b1f elementor-widget elementor-widget-text-editor" data-id="9be7b1f" data-element_type="widget" data-widget_type="text-editor.default"> <div class="elementor-widget-container"> <p><span style="font-weight: 400;">If you’d like to learn more about bringing all of your data sources together, contact one of our data experts.</span></p> </div> </div> <div class="elementor-element elementor-element-e00c794 elementor-button-info elementor-align-center elementor-widget elementor-widget-button" data-id="e00c794" data-element_type="widget" data-widget_type="button.default"> <div class="elementor-widget-container"> <div class="elementor-button-wrapper"> <a class="elementor-button elementor-button-link elementor-size-sm" href="https://www.datalakehouse.io/contact-us/"> <span class="elementor-button-content-wrapper"> <span class="elementor-button-text">Get In Touch</span> </span> </a> </div> </div> </div> </div> </div> </div> </section> </div>