Data Day Edinburgh 2026

Yesterday, 23rd October 2026, was the inaugural Data Day Edinburgh conference. With Data Scotland taking a break this year, it was great to see a local data conference spring up to fill its place. It was a well organised and informative day. There were a wide range of interesting presentations. Here are some of my takeaways from the event.

Cut Costs, Boost Speed: Optimising Your Azure Databricks Lakehouse – Paul McComish

A really good presentation on improving performance and cutting costs on Databricks. Covered basics like

• Prepurchasing databricks units
• Ensuring that all clusters have an automatic shutdown when not required.
• When and when not to use Photon acceleration
• Ensuring delta tables have a regular VACUUM setup
• The automatic optimisations offered by Unity Catalog managed tables

Also covered more advanced topics like:

• Liquid clustering of delta tables and how it is an advance over PARTITION and ZORDER
• Use of OPTIMIZE
• Cluster right sizing
• Delta cache acceleration
• When and when not to use serverless compute
• Genie budgets
• Setting policies in the Governance Hub

One optimisation which was not mentioned which can be useful if you often run deletes or merges on tables is the use of deletion vectors. Overall though this was an excellent presentation.

Beyond Source Control: Using GitHub Projects to plan and execute your data projects – Emilie Rønning

This was a useful introduction to GitHub projects covering how to get started with GitHub projects along with some of its strengths and weaknesses compared to Jira and other planning software. Of particular interest were some thoughts on using labels effectively including using labels to trigger Github actions to populate tickets with boiler plate to reduce errors and save time. I had not known this was possible, so it prompted me to find a tutorial on GitHub Actions for GitHub Projects. Also interesting was ideas such as pulling project data into PowerBI to provide Gantt charts and other information to non technical stakeholders.

AI in the data platform: real work, real risks, and keeping people at the centre – Kellyn Gorman and John Q. Martin

An interesting panel looking at how to approach some common problems when using AI in production. This included:

• The governance gap: What AI tools are active in your estate? What are they being used for? What are their permissions?
• Trade offs between small locally hosted models and frontier cloud solutions
• When using LLMs to prototype remember to check if a deterministic solution or a lest costly more reliable small ML approach would be possible when productionising to control cost and risk
• Ensuring that AI tools do not have different RBAC permissions that the user invoking them. Important in both directions. AI having too few permissions leads to failures and confusion. AI having too many permissions can lead to data integrity and security risks
• Securing AI endpoints and hardening against attacks. The importance of logging and alerting to catch failure modes.
• Considering levels of autonomy for AI models with a suggestion of 4 main levels: Suggesting, Drafting; Acting with Approval and finally Act and Report. It was pointed out that you need better controls and higher level of confidence in the AI’s capabilities to act correctly at the higher levels.

Some useful food for thought which communicated current thought in the field. Interestingly it would appear that we have yet to shake out a totally standard framework for this as a community. The Redgate talk suggested four levels of autonomy to consider. I have also seen similar taxonomies with five levels and three levels of autonomy.

From Individual Contributor to Data Leader: The Skills No One Teaches – Bianca Stratulat

A useful talk about the differences between being a senior individual contributor and a leader. Key takeaways were around the changes in role, for instance

• Being responsible for delivery changing to being responsible for direction
• Moving from doing the tasks to deciding which task should be done
• Moving from the certainty of well defined tickets to responsibility for ambiguous work priorities
• The challenges posed by acting through others and longer feedback loops

These challenges were accompanied by actionable advice on how to approach the new challenges including:

• The importance of delegation and the need to retain ownership when doing so – identify motivations
• Avoiding getting over involved since this leads to becoming a decision bottleneck – make others the hero
• The need to create space for alternative solutions and freedom to fail for your team – lead with curiosity
• Your influence now being tied to repeated reliable delivery by your team – be consistent over time
• Focussing on outcome over output – deliver things which are valuable

The talk also dealt with handling conflict and change with useful advice like

• Naming conflict early so that it can be dealt with before positions entrench
• Understanding that conflict may be as much about priorities as actual differences of opinion
• Creating psychological safety for participants while ensuring respectful disagreement
• Remembering that in the ned you may have to make a call and explain it
• Communicating trade offs upwards early to ensure solutions will be acceptable

Lastly there was a very useful look at change management with advice such as:

• How to identify and handle causes of resistance such as issues of identity, control and fear
• Bringing sceptics onboard early to iron out bugs, sceptics care about the result so they can be great allies
• Ensuring that everyone moves in the same direction but allowing for different speeds
• Ensuring that the legacy experts knowledge passes into the new system by reframing their role from gatekeeper of the old system to knowledgeable expert for the new one

Beyond the Hype: Putting AI to Work, Today – John Martin

This talk started off by pointing out that AI can really be split into three main areas: Building AI, building with AI, and using built in AI. From here It covered some of the key current problems with AI systems: hallucination, confident incorrectness, data leakage, security and jailbreaking.

The talk then delved into the need to align tech, people and process. It also covered some emerging challenges with AI such as cost tracking and value attribution. There was also a survey of the benefits and pitfalls of MCP servers, steering files, knowledge bases.

A couple of neat suggestions were made which I can commend as I have already had success with them:

• When using AI for research, force it to cite supporting sources in such a way that you can independently (and preferably in an automatable way) verify: that the resource exists, is correctly cited, contains the content which is relied on, and does not give rise to contradictions with other authoritative sources. This can extend to defining which citations are allowed. Those from particular online sources? Internal company documentation? Your AI solution’s RAG system? Here is a useful link to how to build a forced citation research agent
• When using AI for coding beyond simple scripts, favour spec driven development and test driven development over vibe coding. That is, break the task up into simple well specified components to a high level of granularity. Ensure that tests are created in a red-green patten. Impose linting and coding standards. AI makes following coding best practise easier and following it makes AI written code better as well as saving tokens through smoother development

What A Data Nerd Learnt by Automating Their Home (Besides Annoying Their Family) – Rob Sewell

A really excellent talk and not only about home automation using Home Assistant and Zigbee devices. The talk demonstrated how home automation setups can fail and drew interesting analogies between these failures and issues seen in commercial software development. The talk ran the gamut from properly defining requirements; through testing, error handling and logging; to high availability, and handling scope creep gracefully. Some really important things to bear in mind when designing complex software systems were illustrated in an approachable and highly informative way. Rob has many other fascinating presentations that you can find here, highly recommended.

Conclusion

All in all this was a fantastic conference. Many thanks to Abhi and Tom for organising the conference. Looking forward to seeing your next effort in spring of 2028. Also exciting is the confirmation the Data Scotland will return on 10th September 2027 at the Technology & Innovation Centre, University of Strathclyde, Glasgow.