OBLYTECH Logo
OBLYTECH Logo

Data Warehousing & Business Intelligence

Why Choose Oblytech for Data Warehousing & BI?

When the same figure appears with different values in different reports, decisions tend to slow down. Oblytech’s Data Warehousing & Business Intelligence (BI) services help your organization consolidate, organize, and analyze your data, transforming it into figures people can act on.

black-man-trader-coach-and-screen-with-stock-mark-2023-11-27-05-19-35-utc

Our Data Warehousing & Business Intelligence IT Solutions

We give your team the tools and training they need to access and analyze data independently, fostering a data-driven culture throughout your organization.

We create customized reports and dashboards tailored to your specific KPIs and business requirements, providing you with the insights you need to make informed decisions.

We help you select and implement the right BI tools for your needs, enabling you to create interactive dashboards, reports, and visualizations.

We design data models and schemas that optimize data storage, retrieval, and analysis, ensuring efficient query performance and data integrity.

We create efficient ETL processes to extract data from various sources, transform it into a usable format, and load it into your data warehouse.

We design and build reliable data warehouses that consolidate data from disparate sources, ensuring data integrity, consistency, and accessibility for analysis.

Decide whether the warehouse or the reporting layer comes first

Organizations that ask for a data warehouse are usually asking for one of three things: reports that finish before the meeting starts, one version of a figure that every department accepts, or the ability to ask a new question without waiting for an analyst to rebuild a spreadsheet. Those needs can point to different first projects. A reporting tool connected to a handful of source systems can answer a narrow set of questions quickly. A warehouse takes longer to build, but it is the place where sources are reconciled, history is kept and business definitions are applied once rather than inside every dashboard. The first decision is therefore about the question the business needs answered, and the technology comes second.

What the business is asking forUsually a sensible first stepWhat that step does not solve yet
One monthly management pack built from two or three systems that finance already trustsA governed reporting model on a small set of sources, with the definitions written downLong-term history across systems that change their codes
Figures for the same customer or order that disagree between departmentsA warehouse layer that matches keys across sources and records which source wins for each fieldSlow dashboards caused by slow source extracts
Managers who want to answer their own questions without a data requestA curated layer with a short list of certified measuresData that nobody has been made responsible for
Trends over several years from systems that overwrite recordsScheduled capture of snapshots before the source loses the earlier stateImmediate charts of trends that were never captured

A useful test is to ask what happens when a number is challenged in a meeting. If the answer requires someone to open three spreadsheets and a query, the business needs the warehouse layer before it needs more dashboards. If the answer is clear and the disagreement is only about presentation, a reporting project is likely enough.

Why the same figure appears differently in different reports

Disagreement between reports is rarely a software fault. It usually comes from three gaps. The first is that the same word means different things in different systems: a customer may be an account in billing, a contact in the sales system and a site in service management. The second is timing, where one report reads yesterday’s extract and another reads a live table. The third is ownership: nobody has been asked to decide which source is authoritative when two of them disagree. A warehouse project forces those decisions into the open because the data must be loaded into one shape. Teams that expect the tool to resolve the disagreement on its own usually find the same argument repeated in the new platform.

The practical work is a short register of shared measures. For each one, the register names the business owner, the source of record, the grain (for example, per invoice line or per month), the rule for handling reversals and corrections, and the date from which the rule applies. A few dozen well-defined measures are more useful than hundreds of loosely described ones, and the register becomes the reference that the warehouse design follows.

How a warehouse and BI build is usually sequenced

Work of this kind tends to follow a sequence whatever the target platform. The first stage inventories the sources, their owners, their refresh patterns and their known quality problems. The second stage agrees the measures and the reporting questions that the first release must answer. The third stage designs the model: the staging area that keeps source records as received, the conformed layer where keys are matched and definitions are applied, and the presentation layer that users see. Loading routines and their error handling come next, then testing that compares warehouse totals with the source systems for the same period. Dashboard design usually becomes the main activity after those steps.

  • Source inventory: which systems hold the facts, who can change them and how often their data is complete.
  • Measure register: the agreed definitions, owners and grain for the first set of reports.
  • Reconciliation tests: totals from the warehouse compared with source totals for closed periods, so that differences are explained before users see them.
  • Access rules: who can see salary, customer or regulated fields, and how those rules are applied in the reporting tool as well as in the database.

Reconciliation is a step that is frequently shortened under deadline pressure, and it strongly affects trust. A pack that is correct in the first week and then changes by a small amount after a source correction can be doubted for months.

Choosing a BI tool without locking the design to one screen

Several established reporting tools can connect to a warehouse, and the choice between them matters less than the model underneath. A tool chosen before the model is settled tends to encourage calculations to be written inside individual reports, which recreates the original problem in a more polished form. The selection criteria that matter in practice are how the tool handles row-level access, whether certified datasets can be shared across reports, how it refreshes, and how easily an analyst who leaves can be replaced. License cost and interface preference should follow those questions rather than lead them.

A buyer should also ask who will build and maintain the reports after the first release. Many projects deliver a good set of dashboards and then stall because nobody has time to add a measure or fix a broken join. Agreeing that ownership early is usually cheaper than discovering the gap after the build team has left.

Where warehouse projects usually go wrong

These failures recur often enough to plan around. Projects that try to load every source in the first release run out of time before the reports are useful. Projects that skip the measure register rebuild the disagreement inside the new tool. Projects that treat the warehouse as finished at go-live discover that source systems changed a field name and the loads began failing silently. Projects that give analysts a powerful tool without a certified layer end up with many near-identical reports, each slightly different. Each of these failures has a visible early signal, such as a long list of unnamed sources, a reporting tool with no shared dataset, or a load job without an alert, and a short review can catch them before they become expensive.

Performance deserves its own check. Query speed depends on how the tables were modeled for the questions being asked, on how much history is kept in hot storage, and on how many users run heavy reports at the same time. A design that suits monthly packs may be slow for daily operational questions, and the cost of changing the model later is usually higher than the cost of asking the question early.

What a buyer should ask before engaging a provider

Question for the providerA good answer describesA weak answer describes
How will you decide which source is authoritative?A method for each shared measure, agreed with the business ownerA general statement that the data will be cleaned
How will you prove the warehouse matches the sources?Reconciliation tests on closed periods, with written tolerancesA promise of accurate reports
What happens when a source changes?A change process with load monitoring and alertsSupport will fix it when it breaks
Who maintains the reports afterwards?A named handover and a documented modelTraining sessions only

The main drivers are the number and condition of the sources, the number of certified measures, the depth of history required and the level of access control. A provider that offers a fixed figure before seeing those inputs is describing a guess.

Questions people ask about data warehousing and BI

Do we need a warehouse if we already have a reporting tool?

Not always. A reporting tool with direct connections is adequate when there are few sources, the definitions are already agreed and the history is kept by the source systems. A warehouse becomes necessary when figures must be reconciled across systems or history must be preserved.

Can the warehouse be built in stages?

Yes, and that is usually the safer route. The first release should cover one reporting area from source to report, with its sources reconciled, so that the team learns the pattern before adding others.

What is the difference between the warehouse and the BI layer?

The warehouse stores and reconciles data over time. The BI layer presents selected measures to users. Problems in the warehouse tend to show up as wrong numbers in the reports built on it, which is why the two are designed and tested separately.

How do we keep sensitive fields restricted?

Access rules should be defined in the measure register, applied in the warehouse and enforced again in the reporting tool. Relying on the report layer alone leaves the underlying data open to anyone with database access.

Next step for a data reporting decision

If the disagreement between reports is costing meeting time or trust, the sensible next step is to write down the sources and the measures that matter most. A list of the sources and the reports that are challenged most often shows whether a warehouse layer, a reporting project or a smaller definition exercise should come first, and it can be described through the contact page.

For the wider context, the data analytics page is the overview of this service area. Teams that are still deciding what to measure may look at the data strategy page, and teams whose sources arrive in messy form may look at data engineering and pipeline development. Where the difficult part is ownership of the source records, the ServiceNow data governance and source authority page covers source ownership for ServiceNow data.

Data warehousing and business intelligence compared with a data lake and operational reporting

Data warehousing is the practice of copying data from several systems into one governed store, organized for questions that span systems and time. Business intelligence is the layer of reports and dashboards that reads from it. A data lake and the reports inside an operational system are neighbours, not substitutes, and choosing between them is mostly a question of who will read the answer.

OptionBest suited toTypical limit
Reports inside an operational systemQuestions about work in progress that one system fully coversUsually cannot reconcile figures held in other systems
Data warehouseAgreed measures, history and comparisons across systemsNeeds modeling effort before the first report
Data lakeRaw or loosely structured data kept for later analysisWithout curation, people build competing versions of the same figure

Many organizations end up with a lake feeding a warehouse: raw files land first, and only the cleaned, defined subset reaches the layer that managers see. That works only when someone owns the step between the two.

A simple rule helps. If the question can be answered from one system and the answer is only needed today, use that system’s own reporting. If the question compares systems, needs last year as well as this year, or must give the same number to finance and operations, it belongs in a warehouse. If you do not yet know the questions, store the data cheaply, but do not call it analytics until someone has defined the measures.

The same rule applies to the cloud. A hosted warehouse removes hardware work, but it does not decide what a customer or a closed order means. Teams moving analytics to a hosted platform may find the big data and cloud analytics page a useful companion. Teams that plan to feed machine-learning work from the same store should check whether the data is ready first, which the AI data readiness page describes. And if the data you want to report on lives in ServiceNow, compare the warehouse route with the platform’s own reporting and dashboards before copying anything out.

Is a data lake the same as a data warehouse?

No. A warehouse stores data that has been modeled and defined for reporting, while a lake usually keeps data in its original form until someone needs it.

Keeping last year’s figures right when a source record changes

A warehouse keeps history by storing each change to an attribute as a dated version, so a report shows the picture as it stood on the date it describes. If a customer moves from one sales region to another and the source system overwrites the region field, every past report grouped by region shifts without anyone noticing. A dated copy of the attribute stops that.

A customer belongs to the North region for the first half of the year and moves to the South region in the autumn. The source now holds South only. Without history, first-half sales by region put that customer under South, which is wrong for the period asked about. A start date and end date on each region assignment let the warehouse answer both questions: where the customer belonged when the sale happened, and where they belong today.

Two decisions follow. The first is which attributes need history. Region, account owner and product tier usually do, because they decide how results are grouped. Phone numbers and email addresses rarely need it, and versioning them adds cost without helping any report. The second is which date marks a change: the date the source recorded it or the date it took effect in the business. These differ more often than teams expect, especially for back-dated contract changes, so the measure register should name the rule for each attribute.

The common failure is to start from the current state of the source and assume history will build itself. It will not. A value overwritten before the first load is gone unless a snapshot was taken, so the capture schedule should start before the first report is signed off, not after someone asks why last year’s numbers moved. Where the source keeps an audit trail of field changes, the warehouse can be seeded from it, but check that trail first. Some systems keep it only for a limited period, and some never record the field at all. On ServiceNow data, the gap often sits alongside CMDB data quality problems.

Test the approach before building. Pick a record that changed last quarter and confirm the warehouse returns the old value for the old period and the new value for the new one. Run that check alongside your reconciliation checks. If it fails, the history rule is not working, whatever the dashboards show. For ServiceNow sources, the reporting and analytics services page covers the reporting side of this work.

ServiceNow guides on this topic

Guides from the Oblytech ServiceNow library that explain the practice behind this page.

More in this section