Erika Pullum

Snapshot Your Metrics Models

Mon, 14 Feb 2022 00:00:00 +0000

Why Snapshot Your Metrics?

Metrics like your company’s financials are important. Your stakeholders have some of these numbers memorized. They trust you to make sure these numbers are right. They’ll notice right away if historical values change when they shouldn’t.

A classic post on the dbt Discourse explains how to use snapshots to ensure your metrics aren’t changing on you. Build Snapshot-BasedTests to Detect Regressions in Your Historic Data was written in 2020 by Evgeny Rubtsov. Back then it required custom tests and a special macro.

I implemented this recently for one of our financial models. I took advantage of ✨new✨ dbt features to avoid writing anything custom. Here’s how it works.

📸 Create a snapshot of your important metric model
🧪 Add a test to ensure dbt_valid_to is always null, meaning your data hasn’t changed
😎 Rest easy knowing regressions will pop up as test failures
🛠️ If failures do occur, update your test after root cause investigation

Fortunately (unfortunately?) the system worked and my test did detect a regression. I was able to take quick action before our stakeholders noticed. I also dug in on the root cause so we can be assured this won’t happen again.

1. Make a Snapshot of Your Model

Let’s say you have a financial model, like sales by month by customer. Your model might look something like this:

month_of	customer_id	sales
2022-01-01	8675	2000
2022-01-01	5309	9000

In this case, we don’t have a unique key we can use in our snapshot. We’ll have to make one. This is quite easy to do with dbt_utils.surrogate_key

-- assume your real model is above this point
-- add a surrogate key like so

select
 month_of
, customer_id
, sales
, {{ dbt_utils.surrogate_key(['month_of', 'customer_id']) }} as surrogate_key
from final

Next up let’s make our snapshot.

{% snapshot sales_snapshot %}

{{
 config(
 target_database='analytics',
 target_schema='analytics_snapshots',
 unique_key='surrogate_key',

 strategy='check',
 check_cols=['month_of'
 , 'customer_id'
 , 'sales']
 )
}}

select
*
from {{ ref('sales') }}
where month_of <= current_date - interval '1 day'

{% endsnapshot %}

Let’s break down what’s going on in the snapshot. We’re snapshotting our sales model, with the check strategy. This means every time the snapshot runs, it will compare the previous values of the columns specified in the check_cols list against their current values.

If there’s a difference, a new row will be inserted into the snapshot with the new values. On the old row, the dbt_valid_to timestamp will be updated to the time the snapshot ran. The newly inserted row will have a dbt_valid_from equal to this time, and a dbt_valid_to that’s null.

We can give ourselves a grace period with the where line in the SQL query.

2. Adding Tests

We’ll see a change in the snapshot model if something changes. Let’s add a test to notify us if this happens. Back in 2020, a custom test was required. Now, dbt_utils has you covered with expression_is_true.

Love that I never have to write custom tests in dbt. There's almost always a way to do it with dbt_utils instead!

Haven't had to use dbt-expectations yet but I've heard lots of great things.
— Erika Pullum she/hers (@erikapullum) February 3, 2022

version: 2

models:
 - name: sales_snapshot
 tests:
 - dbt_utils.expression_is_true:
 expression: "dbt_valid_to is null"

If our metrics change, the snapshot will insert new rows. The test will fail. We’ll know a problem has occurred an we can investigate and determine what caused it.

The test failing tells us that something changed in the source that shouldn’t have. This is good, because we’ll know about the problem. We need to fix the test though, because a failure every day could cause us to miss something else that’s important.

3. Updating the Test After a Problem

Let’s say one of our sales numbers changed, for customer_id 8675. The snapshot will now have two records for that customer:

month_of	customer_id	sales	dbt_valid_from	dbt_valid_to
2022-01-01	8675	2000	2022-01-02	2022-02-14
2022-01-01	8675	3000	2022-02-14	null

Our test will be failing due to the first row in the table, and we’ll get a message something like this:

06:10:53 Completed with 1 error and 0 warnings:
06:10:53
06:10:53 Failure in test dbt_utils_expression_is_true_sales_snapshot_dbt_valid_to_is_null (snapshots/snapshots.yml)
06:10:53 Got 1 result, configured to fail if >0

Since we’ve understood the failure mode already, we need to fix the failing test. It has done its job by alerting us – we don’t want it failing every day!

One method for fixing the failing test is to remove, edit, or update the rows in the snapshot. Since dbt 0.21.0 severity configurations mean we can get the test back to passing without modifying the snapshot table.

Severity configurations let us specify a severity of error or warn for a test. They also allow us to specify the number of result rows the test must return to be considered failure.

This error message gave us exactly what we need to get the test passing. We can update the severity configs so that the test is passing again.

version: 2

models:
 - name: sales_snapshot
 tests:
 - dbt_utils.expression_is_true:
 expression: "dbt_valid_to is null"
 config: #update when exceptions occur
 severity: error
 error_if: ">1"
 warn_if: ">1"

This requires a PR, which is probably a good thing! This gives you an opportunity to document what caused the failure, what corrective actions you took, and why the change to the test is needed.

What Should You Snapshot Like This?

This strategy is effective, but it takes work to maintain it. What types of models is it good for?

Models that cannot be wrong, like financials or key KPIs
Data that’s aggregated by a date dimension, like a week or a month
Metrics that shouldn’t change after a certain amount of time

Have thoughts? Find me on Twitter and let me know!

Analytics Engineering Dotfiles

Mon, 29 Nov 2021 00:00:00 +0000

What are dotfiles, anyway?

Dotfiles are configuration as code for your development environment. The name originates from files like .gitignore or .zshrc, which start with a . and became referred to as dotfiles.

One of the cool things about dotfiles in the software world is that they’re frequently shared. You can learn from what other people have done. I was introduced to dotfiles as a concept by a software engineering colleague of mine.

There don’t seem to be many examples of analytics engineering dotfiles out there. What I did find is usually called “setup” and not dotfiles¹.

Here are some examples:

To Fork or Not to Fork

It turns out there’s a whole debate on whether or not you should fork someone else’s dotfiles. Forking in this context means copying someone else’s approach for your own environment.

If you’re new to working with dbt, my dotfiles might make a good starting point. However, dotfiles are highly personal. What works for me might not work for you. Even if my exact setup is perfect for you, you’d need to learn what I know to get the same efficiency I do. Your pain points might not be the same as mine, and things I love might not work for you at all.

With that said, please do fork away! I tried to do a good job documenting what’s going on in comments and on the repo. I’d love to see pull requests, issues, and contributions to this repo. We’re still in the early days of analytics engineering as a discipline. I don’t think we can share enough.

I’m just getting started with customizing my command line environment. I’m sure there’s a lot that could be improved here. Like Stone Soup, I’m hoping that by putting this out in the world, people will be moved to show up and put something in the pot.

Erika’s analytics engineering dotfiles on GitHub

A Little Bit of Erika by Your Side²

Here’s some of the things the dotfiles do that I think are pretty neat.

🍺 Manage installed software and packages with Homebrew
🦾 Optimize VSCode for working with dbt by installing lots of good extensions
👩‍🔬 Clone a dbt project so you can easily test your setup on a new computer
✨ Add aliases for common dbt CLI operations
🔎 Make validating data for dbt PRs easy with dbt-audit-helper automation
🤘 Make adding to your dbt project easier with shell script customization for dbt-codegen
🧹 Basic SQLfluff setup for dbt to make linting SQL easy

Making the Dotfiles

I wanted the dotfiles to be simple and approachable. By approachable, I mean that I wanted them to be easily understandable to someone with an analytics background³ and solid dbt experience.

I balanced my desire to have this be general and useful with the need to keep my own configurations centralized. I chose tools that are free to use. For now, I’ve chosen not to bring in a dotfile utility.

I kept things simple, until I couldn’t. For example, I started with all the files in one root directory. I thought people might have trouble figuring out what was what, so I added some documentation. It still seemed confusing, so I added subdirectories. That inspired me to add READMEs for each tool’s configs.

I’m sure things will continue to evolve. There are some tools I’ve used, like Terraform, that I’m not currently working with and didn’t include for that reason.

About Configuring Your Shell

I found shell configuration intimidating and procrastinated on it for a long time even though I use the command line a lot.

It's amazing to have automation for common tasks, even stuff that saves a few keystrokes adds up throughout the day.
— Erika Pullum (Swartz) she/hers (@erikapullum) October 28, 2021

Shell configuration has been life changing for my daily workflows. I wish I’d done it sooner. In the course of building the dotfiles shell scripts I learned some nifty new command line tools.

I highly recommend investing time in learning the ins and outs of the tools you use every day. A little bit goes a long way.

Sometimes it feels like tooling and practices in the data world are a few years behind the software engineering world. Some of that is terminology, like dotfiles vs. setup. However, the problems data folks are solving are different and the data tooling isn’t as good as it should be - yet. ↩︎
Yep, that’s a Mambo #5 Joke. Hope it’s not stuck in your head now. ↩︎
I realized after I wrote most of this post that some concepts in the dotfiles might not be intuitive to the person I’m describing if they haven’t come across them before. For example, someone who has mostly worked with dbt Cloud might not be very familiar with shell customization. Trying to explain everything felt like it would overload this post. If there’s something you’d really like to see me write more about, let me know on Twitter! ↩︎

Upskilling Analysts

Sun, 17 Oct 2021 00:00:00 +0000

A few weeks ago, I got curious about how organizations can intentionally retrain analysts for data science roles.

This post is the result of a few conversations with data leaders who have been there, done that, and my own research on the topic.

Has anyone intentionally retrained analysts into jr DS roles as a strategy in their org?

Can I pick your brain about it?

Saw a case study on the topic that piqued my interest, along with the convo I had with @sethrosen the other day.
— Erika Swartz (@eswartz) September 4, 2020

Who Are You Calling Analyst, Anyway?

When you hear the word analyst, what kind of role do you imagine?

The actual job duties of someone with a title that contains the world analyst have an astounding breadth.

Analyst capabilities can be divided into three broad areas:

business acumen
domain expertise
technical skills

Analysts use these capabilities to make decisions, help others make decisions, or solve problems for their organization.

The diversity of analyst duties is important to think about when you think about retraining analysts to be data scientists. Data science may not be an end goal for every analyst, and the learning curve will be steeper for some analysts than others.

Q: To get into data science is an Analyst role a good starting point?

A: (from a hiring manager, not me) when hiring, it's a red flag. With Data Sciencey stuff on a resume but applying for an analyst job, it seems like it's a stepping stone. We want folks who want to be analysts
— Seth Rosen (@sethrosen) September 3, 2020

Considerations for Data Leaders

This post is for leaders who want to increase their organization’s data science capabilities or output by upskilling analysts.

This post is not about how individuals retraining themselves for data science roles, but I think y’all are awesome, too!

To get your program off the ground in the first place, you’ll need to answer some key questions. Each section below addresses a strategic consideration you’ll have to assess as you design and pitch your program.

Why Reskill Analysts?

Analysts are already insiders. They have hard-won understanding of your organization’s data, knowledge of its business processes, and the trust of its decision makers.

The best analysts can listen to a business problem and a list of wants, and instead of just delivering data, solve the heart of the need, delivering a quick and accurate answer.

The business acumen and domain expertise analysts have gives them a running start on solving data problems. The problems that come their way are a good representation of questions that are hard to answer within your organization, which is useful for finding potential data science projects.

Domain expertise allows analysts to communicate concepts in ways that decision makers can understand and relate to. They know what concerns are likely to come up, and will be able to answer them.

Great data scientists often have some or all of these competencies, too. For the data leaders I talked to, leveraging the experience of analysts was an intentional strategy to speed up the scaling up of their data teams.

How Will You Retrain Your Analysts?

In running an upskilling program, you need to identify the skills you are expecting your analyst team to acquire, and provide resources.

Data leaders I spoke to had a variety of strategies, platforms, and resources from formal training to supported self-study.

Best practices:

Carefully consider the relevance of any training you assign
Ensure a smooth ‘hello world’ experience
Great documentation makes learning easier
Create space for daily practice
Assess progress to identify deficiencies and provide individual coaching
Give analysts the chance to practice and reinforce their skills on real projects

Other considerations:

Who do the analysts report to while they are retraining?
What incentives are in place to encourage learning and reward those who upskill?
How will you create safety if an analyst chooses not to continue?
How many analysts can you retrain at a time? Will you do multiple waves?

So I Don’t Need Data Scientists?

Nope, you need data scientists, too.

The leaders I talked to hired data scientists, leveraged internal talent, or acted themselves in this capacity — to mentor and work with the upskilling analyst team. If you are not the talent, you’ll need to justify hiring it or borrowing it from another part of your org.

Data scientist mentors to the analysts played important roles including:

Mentoring on technical skills unique to data science like:
- Building and testing data pipelines
- Feature engineering
- Model selection and evaluation
Helping analysts navigate new platforms and tools
Pairing to work on a project
Helping connect the dots for concepts or ideas
Encouraging and supporting skill development

Support of experienced practitioners is critical for upskilling analysts to reinforce their learning by applying their skills.

But is it “Really” Data Science?

And do you care, if you’re helping the company increase ROI or satisfy customers?

Data science is a discipline that has been highly and widely hyped for years. Business leaders will have preconceived understandings or misunderstandings of data science. These can work for you, if an outstanding business result can be achieved with a technically simple and straightforward solution. It can also work against you, if a business leader expects data science to deliver miracles, and has little patience for projects that take time to impact business results.

With business leaders, clarify expectations early and often. Engage them in decisions about project direction and ensure they understand the nature of data science.

Consider where your org is on the maturity curve and the motivational fit of the talent you assign to each project.

An upskilling analyst may be incredibly motivated by a simple data science solution that wins widespread praise. An analyst that feels pressure to learn too much, too soon (to keep their job) may be burned out by a high profile project, even if they are successful. A trained data scientist may be frustrated that no one is interested in a more complex solve or bored because they aren’t technically challenged.

Can You Manage Your Stakeholders?

Leading technical functions is a balancing act.

It’s especially challenging in mature organizations, and more change management is needed for new-to-your-org capabilities like data science.

You need to deliver wins for the business leaders who will sell the value of your team laterally and upwards in your organization. Data leaders need to invest time into building relationships and trust with these leaders. Calibrate expectations and involve stakeholders in deciding which tradeoffs to make.

At the same time, you need to keep technical debt low, ensure your team is doing quality work, and keep your team engaged.

Leaders who balance these competing priorities of speed and a happy data team will deliver quick wins and sustained successes.

Still thinking about some advice from my last 1:1 from my boss:

"It's not the best idea or the best analysis that wins, it's the best relationship."

Is this the reason so many #DataScience and #Analytics projects fail?
— Erika Swartz (@eswartz) August 4, 2020

Data leaders focused on getting wins that mattered to stakeholders. Your team’s conclusions need to be trustable and communicated in language that makes sense to your business leaders.

In some cases, a few high profile wins were enough to build the budding org’s reputation. In each case, the data leader could clearly connect the win to a financial measure for the company.

What Will Your Org Look Like 3 Years From Now?

The final consideration is long term. How will you retain the talent you’ve just upskilled? As your org grows, who can step up into management roles? How do you define success for your team?

Contributors

Samantha Zeitlin and I overran our planned one hour chat because she had so many great insights and nuggets of wisdom for me. You can find Samantha’s writing here, several of her posts address this topic.

My deepest thanks to the other data leaders who were generous with their time and ideas, but who did not choose to be named in this post.

Management Seat Time

Tue, 01 Jun 2021 00:00:00 +0000

photo source

In case you haven’t heard, there’s a (r)evolution going on in the modern data stack. There’s a new way of working in data, where software engineering best practices are the way the data team gets work done. I recently left my role as the manager of a large team to join Aula Education as a Sr Analytics Engineer — primarily to get in on the technical fun.

Management and engineering are fundamentally different career tracks. Managing has been a great privilege and I’ll always be grateful to my former team and for the time we had together. This post is a reflection on my years in management and the lessons I learned from them. It’s all about seat time.

Seat Time

Before I became a manager, I applied for a manager role in my department and didn’t get it. The candidate who was selected already had management experience even though they had no experience in my department. I was a high performer in my individual contributor role and I felt disappointed and passed over.

A mentor who helped me understand the decision explained that what he called seat time mattered – and the other candidate had it already. It’s hard to understand what seat time is when you’re young and ambitious and smart. It’s impossible to see the value in seat time when you don’t have it, which is even more frustrating.

Personally, one of the hardest parts of bridging from technical IC into management and leadership has been the unknown unknowns.

Before you bridge, it's hard to know for sure what skills you'll need, and there's no definitive resource for picking them up once you "arrive"
— Caitlin Hudon 👩‍💻 (@beeonaposy) February 26, 2020

For me, seat time has come to refer to the skills and experience required for management, which can only be picked up by doing the job. No amount of reading or thinking can give you seat time. It’s purely experiential.

Self-awareness and introspection are essential to benefitting from seat time. Mentors and a good boss can help you learn more quickly and avoid some mistakes. Time is an essential component of this kind of learning. You can’t shortcut the process.

You can read about these skills in management books— I’ve included some I like in the footnote.¹

The thing is, reading about how to fire someone is different from doing it. It’s easy to read about the need for clear feedback and nod along, but hard not to soften feedback to someone you like at the moment you’re giving it. The impact of your decisions plays out over a longer timescale and it’s not always easy to see your own mistakes.

Seat time is sitting across the table from someone, listening to them describe a situation, and realizing you’ve seen this pattern before. You know how to navigate the situation — even or especially if it involves tricky interpersonal dynamics. When you come across a situation you haven’t seen before, it takes more time and energy to handle it than if you’ve seen variations of it several times.

It’s not that the emotional work of managing gets easier with seat time. You do get better at compartmentalizing and processing the emotional challenges of management.

In the last year and a half or so, I managed a 12 person team of direct and dotted line reports. One cool benefit of management seat time is seeing patterns play out. You see more than just your own work and gain a lot of perspective and understanding from seeing what’s going on with your reports. I learned two key lessons from my seat time, the first of which was …

Not Everyone Is You

That’s a good thing.

This is the seat time lesson I learned, re-learned, re-learned, and learned again in a new way. You can’t treat everyone like yourself, because not everyone on your team is like you! Most seat time lessons sound painfully obvious when you write them out like this.

It’s easy to slip into giving feedback based on what you would have done or thought about, forgetting that your team member does not have the same experiences and context that you do.

A closely related lesson is that you can’t treat everyone the same way. It seems like it would be fair or even right to do this, but it’s not. People have different needs depending on what they’re best at or struggling with, where they are in their career, where they’re trying to go, or what’s going on with them at a specific point in time.

You can’t praise people only in the way you want to be praised. I’m highly intrinsically motivated — if I know I did a good job, that can be enough for me. I had to intentionally understand and adapt to praise team members according to their needs and preferences. Some people need more regular and consistent positive feedback to do their best work.

Coaching and growing your team is one of the most important duties of a manager. I couldn’t coach effectively until I learned to tailor my approach to each individual and what they needed. I had to continuously improve my approach, and focus on a long term perspective.

People sometimes tell you what they need based on how they see themselves, which can be different from how they behave and respond. Listen to your people, but trust your own instincts too.

For example, a team member once told me that they welcomed and wanted direct feedback. A few months later, one of their projects got off track, so I gave them some direct feedback. Then things got worse. The team member shut down, became argumentative, and the project got further off track. What went wrong?

This person had told me what they needed based on how they saw themselves — which wasn’t what they really needed. I kept on giving direct feedback, which was making the situation worse, not better. I wish I had paid more attention to how this person was responding and adapted my approach sooner.

Sometimes managing can feel like balancing while walking a tightrope. You can’t get it right every time, or all the time. If you’re lucky, you have a good foundation of trust and understanding between you and your team members for when you don’t get it right.

This is why seat time is so important. My intuition for when to change my approach got better over time. I improved my ability to context-switch and treat each person as they needed to be treated at that moment. I got better at giving myself grace when I didn’t get it right.

It’s Not Just What You Do and What You Know

This second seat time lesson is one I’m not sure I would count as “learned” yet. Some part of me still wants to believe success is about getting shit done.²

"it's not enough to just be good" would make a great book title https://t.co/vZkYU6ddT8
— Matthew Hawthorne (@mhawthorne) May 3, 2021

One of your jobs as a manager is to build a network with other managers and teams to get things done. You must build relationships outside your team for your team to succeed. Management success is not about what you, personally, can do. It’s about what you can get done through and by others.

The book Being the Boss addresses this aspect of management, which is left out of a lot of other books about management. You can’t solve problems bigger than your team without a network of friends and allies within your company. It’s important to cultivate this network purposefully and intentionally.

Many people become managers because they Want to Change Things™️. It can be frustrating to realize that you can’t change as much as you thought you could. Having a good idea is not enough. Most problems in the organization exist because there is a structure keeping them in place or something is preventing the part of the company that’s causing the pain point from feeling the pain.

This is a part of managing I found particularly difficult in my last role. The organizational and cultural alignment for the changes I wanted to see just wasn’t there. The changes I wanted to see weren’t going to materialize.³

Benefitting from Other People’s Seat Time

If you are a manager you need advice on how to manage — you need someone you can call. Someone other than your boss, someone you trust, someone who will call you on your BS, and someone who has more experience than you do. A lot of my regrets my time as a manager come from situations where I should have asked for help and didn’t.

I’ve collected some mentor tips that helped me:

Hope is not a strategy, ma’am.

You can’t want it more than they do (on coaching direct reports).

Don’t waste emotion.

It’s not the best idea that wins, it’s the best relationship.

Listen. Ask Questions. Ask questions you know the answer to. Check opinions. Don’t assume.

Know your 1-2 key messages in any presentation.

There’s life outside of your company.

I heard this final bit of advice when I was in the process of leaving my management role for a technical role. I had been job searching and interviewing and I felt sure I was making the right decision.

Great leaders always have three things – aptitude, vision, and passion. Vision means you’re excited, you’ve got a dream. It doesn’t matter if it’s right at first, just that you’ve got it. With passion, no one can light your passion for you and no one can put it out. If your passion’s out, it’s time to go.

I’ll always love helping people grow and develop. My passion for management had gone out and I dreamed of learning and applying new technologies. It was time to go.

Here’s are some management books I like or have learned from - True North, Radical Candor, Being the Boss, No Hard Feelings, Can’t Hurt Me, Peak Performance, The Making of a Manager, The Manager’s Path. I’d love suggestions for more to read in the comments! ↩︎
Matthew recommended the book “Stealing the Corner Office” which was an unsettling and interesting read. Writing unsettling reminded me of this article which I was able to find instantly because I had it labelled on my reading list as “extremely unsettling” ↩︎
Big thanks to Charity Majors for a footnote in a blog post — “And if you’re going to ask should you quit your job, I will save you a phone call: yes.” For some reason reading that allowed me to give myself permission to walk away and not try to fix those things. ↩︎

Learning is a Superpower

Mon, 01 Mar 2021 00:00:00 +0000

I wrote the first version of my reading list digest in 2019.

I’d transitioned from a role in Operations to a role in Analytics. There were so many things I didn’t know, and so many I wanted to learn.

In the course of searching for the answer to a problem, I’d find something cool or interesting. When this happens, I want to stop solving, pull over, and learn something new.

This is not a good way to get things done at work.

I started saving articles to a spreadsheet creatively titled Reading List. I like to track things, so I added some columns for when I added the article, when I read it, and what I thought about it.

This evolved into a personalized, automated daily digest of articles to read. If you want to skip reading about how it came to be, you can check it out and create one of your own here.

Make It Easy, Not Hard

Anything you want to happen goes better when you create a system that makes it harder to mess it up.

I learned this lesson first in my career in operations. Processes shouldn’t expect people to be perfect. Simple over complex. Automated over requiring judgement.

In one case, we traced a problem back to a 14 point checklist which we expected operators to execute, perfectly, at 4 in the morning, in a noisy environment wearing PPE, after 9 hours of a 12 hour shift. We changed the process. It never happened again.

I can think of dozens of examples like that from my time in operations. Internal company software is rarely as intuitive as it should be¹. Don’t expect perfection from yourself or from people using the things you build.

Make systems that make it hard or impossible to mess ‘it’ up.

Dan John², one of my favorite coaches, calls habits that make things easier Shark Habits - One Bite and Done.

Shark habits save time. Shark habits save mental overload.

An important point: Shark habits don’t judge whether or not something is important, unimportant, trivial, or the key to life, living and the universe.

Automating my reading list is a shark habit that enables me to learn.

Automate the Decision of What to Read

At first, the spreadsheet worked great.

I had so little on there that I could remember from the URL which article was which and what I wanted to read most.

Then the backlog grew.

Then it grew some more.

Suddenly reading was a chore instead of fun. I’d save articles up for a big batch when I had downtime and was in the mood to read. I’d also go days or weeks without reading anything at all.

I keep a spreadsheet for articles I want to read, talks I want to watch, or resources I want to look at.

I wrote a little script to randomly choose 1-3 items off the backlog and send me a digest email, and since then I've been reading (and tweeting) a lot more! pic.twitter.com/2N0GDBU3KX
— Erika Pullum (Swartz) she/hers (@eswartz) January 9, 2020

I realized part of the problem was that I had a hard time deciding what to read.

So I automated the choice away from myself. The email comes, I read what’s in it. One bite and done.

How It Works

Google Apps Script is a javascript based language that allows you to perform automation from a spreadsheet. I use a spreadsheet of articles with statuses as the base.

The script I wrote looks for all of the unread articles
It alternates between sending the oldest article and one of the newest 20 articles which keeps the digest fresh
It will send between 1 and n additional unread articles per day, where n is a maximum you can set

Where I Find My Favorite Reads

Late spring in 2019, I had a mentoring meeting with a senior leader at Shaw. I didn’t see myself leaving Shaw, but I worried I would stop learning or get too used to how we do things. The leader advised me to go to conferences to get external perspectives.

I’d just left an operations role with 30-50% travel and I was burned out on being away from home. And I’m an introvert. And conferences are expensive, and (at the time) required travel. So I got on twitter instead.

I started by looking for interesting people writing about analytics. I’d be looking up something, like ‘what’s an index?’ find a cool article, and go track down the author and follow them.

Twitter created another problem. I’d find cool articles messing around on my phone when I was nowhere near my computer.

So I started emailing them to myself, so I wouldn’t lose track of them.

Eventually I added more automation, to scrape emails out of my inbox and add them to the reading list.

On Learning and Being Uncomfortable

It took me a long time to finish generalizing the code and publish this writeup.

Part of that was just life — but when I sat down to write this post, I realized part of it was fear.

Fear that the code wasn’t good enough — I taught myself javascript and it’s not my best language.

Fear that people would think this whole thing is dumb.

Fear of not having the “right” list of articles to put out there.

Fear that other people might judge what I consider list-worthy.

Fear is the biggest blocker of my learning, and probably yours too.

You can follow all the tutorials, watch all the talks, and read all the books.

In the last year and a half, I’ve read 306 articles. I read articles I agreed with, and articles I disagreed with. I read about technologies I may never get to use myself. I read writing by wildly successful leaders and thinkers.

There’s no substitute for the learning you gain by moving through discomfort and doing the thing anyway.

My reading list informs my thought process and approach. However, the biggest lessons I learned haven’t come directly from it.

They’ve come from taking risks, getting uncomfortable, from mentors, from my team, from making mistakes, from reflection on what didn’t work. If you’re not uncomfortable or unsure from time to time, you’re probably not learning.

My digest data, from a year and a half or reading

Citibank got an expensive lesson on the importance of UI design recently. This writeup includes a screenshot of the actual screen where the mistake was made. ↩︎
Other favorite Dan John isms are “Keep the Goal the Goal” and pirate maps. I first heard Dan on the Power Company Climbing podcast in 2017. ↩︎

Deep Work for Data Teams

Tue, 15 Dec 2020 00:00:00 +0000

There’s a particular kind of work that’s hard to appropriately prioritize — things that are important but rarely urgent.

Here are some examples for analytics teams:

None of these activities are important and urgent, as long as things are going well for your team. If you neglect them for long enough, however, your team and your internal customers will feel the impacts.

Documentation, for example, is easy to bump from one day’s to do list to the next. It’s rarely important enough to stop everything and prioritize. If it’s not done well, the team will see the impacts in slower onboarding, trouble learning new skills, slow transfer of knowledge, and failure to leverage new tooling or methods.

It’s easy as a leader to say that your team is empowered to make time for learning, documentation, or other important but not urgent work. In practice though, we noticed that we were struggling to actually do these tasks, week in and week out. Deep work blocks started as a one day trial that we’ve scaled up into a larger experiment.

What is a Deep Work Block?

Deep work blocks are a great solution to schedule the important but not urgent work to make sure it gets done. The social aspect of doing them as a team provides built in accountability and prevents urgent tasks from sneaking in.

A solid block of time — a half day or a full day — provides the focus that makes deep work effective. You have time to think deeply, get lost, get off track, find your way, and get uncomfortable. Deep work is effective for activities where you have to learn a new tool and then use it. These are hard to accomplish in slices of time between meetings.

A deep work block should be focused on a particular objective that relates to something the team needs. This focus enables the team to improve or accomplish something quickly, which creates a delightful sense of momentum.

Deep work blocks are most effective when they’re collaborative, long enough, and focused.

It's deep work day for my team! Digging this experiment template I stole from @johncutlefish

Today we're practicing Python skills! pic.twitter.com/l32TIfDnmm
— Erika Pullum (Swartz) (@eswartz) December 1, 2020

A Data Manager’s Playbook

This section is for managers who want to try this tactic with their team. Read on below for how to prep for a successful deep work block.

Explain the Why

Connect what you’re doing in the deep work block to the mission and values of your team. It’s important to articulate that this work is still meaningful and explain how you want the team to engage with it.

This is especially important for unexciting deep work, like updating documentation, adding tests, or cleaning up technical debt.

Test Drive the Tools

Whether it’s you or someone else, test drive anything the team doesn’t yet understand to ensure it fits your needs and goals. Prepare the ‘hello world’ experience of your deep learning day carefully, especially if it requires new tools, new conceptual understanding, or a new process.

For example, we used the Elements of Data Science Google CoLab Notebooks by Allen Downey for our Python deep work day. Before we committed the whole team to it, I had the analysts with the least amount of Python experience test drive a few of the notebooks. This also gave them a head start the day of deep work, since one of our goals was to bring Python experience up across the board.

It was not an accident that the activity I chose didn’t require installing Python to start. I wanted to ensure that getting started was smooth, easy, and didn’t involve troubleshooting.

Clarify Expectations & Define Success

Set it up so that the team understands what you expect. How open ended or exploratory is the effort? Can you team expect to show up and follow directions for some task-oriented or learning oriented deep work? Or is the goal to step back, reflect, debate, and discuss? Do you expect to leave with a deliverable or other outcome?

Making decisions by committee doesn’t work very well. As the leader, clarify your expectations up front about how the day should go.

For our documentation day, I gave each team member a folder of documentation to review and update, and a goal of getting it done by the end of the day. For our learning oriented day, the goal was just to make progress in the allotted time.

In developing new team processes, having a large team debate it out might not be a good use of everyone’s time. In that situation, I like to task one or two team members with producing a strong first draft. I tell them what I want to see addressed. The deep work block is then used to strengthen the draft with the rest of the team, and create buy-in for moving forward.

Kick Off and Close Out

On the day of, kick it off together as a team. Remind everyone of how this work connects to your needs as a team. Clarify ground rules like when to ask for help and where. Build some time to reflect into the exercise, whether it’s the day of, when everything is fresh, or later in the week. Seek feedback by asking open ended questions and being prepared to hear candid feedback about what didn’t work.

A Data Team Member’s Pitch

What if you don’t lead a team, but you still want deep work? Here’s a short guide for what to think about before you pitch this to your manager or team lead.

What do you want to work on?

This part is pretty easy — what’s inspiring or exciting enough that you want to spend time learning it?

What’s a pain point you know the team could get rid of, if you invested some time in making things better?

Write your thoughts out or explain it out loud until the concept is clear and concise.

Why is it important?

Think about what you know about your team’s direction, strategy, and challenges. Ideally your proposed deep work connects to something your manager or team lead cares about, whether that’s something they want your team to start doing or do better.

Create a clear connection between a team goal, and be ready to explain how the deep work you’re proposing will help. Ask ‘5 whys’ to make sure you’ve thought through the connection as deeply as possible. Come up with a few examples of potential wins to share with your manager.

Elevator Pitch

Finally, build out your elevator pitch. Think about the two points above. Consider practicing on a peer before you try it out on your manager. If your organization or team already has a process for proposals, consider preparing the first step in the process for review with your manager.

If your manager or lead isn’t immediately receptive, take a step back and seek to understand what you’re missing. What are their concerns? Don’t be afraid to incorporate feedback and attempt a new pitch!

Have feedback? I’d love to hear it!

Continue the conversation on Twitter

Improving Onboarding for Data Teams

Fri, 01 May 2020 00:00:00 +0000

Starting a new technical role can be a frustrating pendulum between boredom and overwhelm.

Getting set up takes time, and once you are set up, you’ll have lots of questions as you get used to new tools, data, and codebases.

I'm working on some onboarding materials for our DS team. Besides the obvious, HR type things, what technical docs are you looking for as a mid-senior level data scientist when you walk in the door at new job?
— Stephanie Kirmer (@data_stephanie) April 22, 2020

On my team, we’ve developed a process to keep new hires moving and learning. We use 3 key documents to help coordinate the team to bring our new member up to speed.

They are:

An access / skills checklist
An onboarding journey doc
Our overview of how we organize our documentation

Access / Skills Checklist

This document is the workhorse of the three. It provides a week-by-week road map of tasks and skills. For this demo, I’ve generalized it for a fictional new associate, Jane Smith.

Colored squares identify where a skill is - need help, need practice, or solid. I use initials to indicate who is responsible. Sometimes I need to initiate something and then after that has occurred, the new hire will need to follow up.

A blue dot indicates something we know needs better documentation. Maybe there are directions, but they’re not up to date. Maybe there aren’t any written down yet. Once the new hire learns the skill, they’ll get initials in blue to indicate they need to create the documentation or fix what’s already there.

We’re on our third or fourth iteration of this list - every time we hire someone new we find something missing or something we could do better.

What works well

Every new hire refines your documentation
Give new hires a clear road map, week-by-week, for getting capable in the role
There’s something to do when peers / collaborators aren’t accessible - go get more green on the list
Balance learning styles by offering options & supporting learning by doing

What needs an upgrade

A checklist does a bad job communicating concepts
My ‘working’ doc is still missing the “why” column. The original did a great job telling you what to do but didn’t tell you why it mattered
As work picks up, new associates shift away from learning towards doing. I need to find a better way to create accountability through the entire onboarding process.

Onboarding Journey Doc

Checklists lack soul. The onboarding journey document is personalized by me for every new associate. It breaks down into five sections.

Key Reference Links

New hires are learning a lot very quickly. This list of links has the access / skills doc and a couple other key internal documents. My goal is to help associates quickly find help. I add links to this section as the new hire asks questions or gets stuck.

Peer Support Resources

I identify 1-2 key team members who will support the new hire’s learning and answer questions as they pop up.

Week by Week Schedule

This section ties the access / skills list to the onboarding journey. Weeks 1-2 are all about basic access, weeks 3-6 are about building core skills for the role.

One of the key ways we do this on my team is by having a new hire cover 100% of support requests for 6 weeks. When they cover support, they meet business partners, build a picture of our existing portfolio, and sort through any access issues that crop up.

By the middle to end of this time, an obvious first project has usually come up. With appropriate direction and support, my recent new hires have been executing on projects within 6 weeks.

Key People to Meet

In this section I build a big list that takes about 3 - 6 months to work through. It’s customized to each associates’ role and prioritized by their key business relationships. Depending on the person’s role, I will ask the new hire to either meet or job shadow the person.

Potential Projects

The team helps me build up this list with good first projects. We want our new hire to get some wins early to build their confidence and get them engaged.

What works well

Personalizing the document to each new hire’s needs and experience
Providing a clear week by week road map
Most are happy to see a “menu” of projects they could work on
Maintain flexibility to adjust as things go on

What needs an upgrade

In practice, we stop using the doc once project work picks up
To fix this, I could use the onboarding doc as the 1:1 doc
It’s hard to keep all the different documents straight - new hires need one central landing place

Documentation

Good documentation is essential for a high functioning collaborative team. Our final doc new hires get is a living document which explains how our team creates and uses documentation.

Each team has to come up with its own norm for documentation - where you store it will be different from company to company and team to team.

In general, this document should cover all the bases and evolve over time to be as clear and helpful as possible.

Here are the guidelines for ours

If documentation is wrong or unclear stop what you are doing and fix it
If you don’t know how to fix the documentation, ask the team!
The folders are numbered and order by priority
Put the documents in the most logical place from the list below
All meeting notes and project files should be dated
Use YYYY/MM/DD as your date format so files order by date with the folders
The last guideline covers what documentation is for team members (internal) vs. externally facing (within the company)

My team is reorganizing our collaborative documentation drive.

The only topic with 100% agreement?

Date things YYYY/MM/DD at the start of the file name so they sort. pic.twitter.com/4xqxUuAdZY
— Erika Swartz (@eswartz) October 3, 2019

What works

I can find documentation where I expect it
This reduces everyone’s need to interrupt each other to ask for information
We built the content strategy collaboratively

What needs an upgrade

Keeping up good documentation is hard
Documentation can be confusing if there’s not enough context about when it applies

How will you know it’s working?

Onboarding is hard. Doing it well requires practice, feedback, and iteration.

My goals for onboarding a new team member are simple:

I want them to feel welcome and valued from day one
I want them to have help when they need it
I want them to have support and space to learn
I want them contribute to the team and our work in ways that feel meaningful and interesting to them

Do all that well, and value for your company and your team will follow naturally.

As a side note, this framework works for documenting your current job too. It’s especially helpful if you’re promoted internally to prevent questions related to your old job during your transition period. The access / skills list would stay the same, the onboarding journey doc would focus on in progress initiatives, and the documentation about documentation address how you’ve documented your own work.

Erika Pullum

Snapshot Your Metrics Models

Why Snapshot Your Metrics?

1. Make a Snapshot of Your Model

2. Adding Tests

3. Updating the Test After a Problem

What Should You Snapshot Like This?

Analytics Engineering Dotfiles

What are dotfiles, anyway?

To Fork or Not to Fork

A Little Bit of Erika by Your Side2

Making the Dotfiles

About Configuring Your Shell

Upskilling Analysts

Who Are You Calling Analyst, Anyway?

Considerations for Data Leaders

Why Reskill Analysts?

How Will You Retrain Your Analysts?

So I Don’t Need Data Scientists?

But is it “Really” Data Science?

Can You Manage Your Stakeholders?

What Will Your Org Look Like 3 Years From Now?

Contributors

Other Reading on This Topic & Things That I Read Along the Way

Management Seat Time

Seat Time

Not Everyone Is You

It’s Not Just What You Do and What You Know

Benefitting from Other People’s Seat Time

Learning is a Superpower

Make It Easy, Not Hard

Automate the Decision of What to Read

How It Works

Where I Find My Favorite Reads

On Learning and Being Uncomfortable

Deep Work for Data Teams

What is a Deep Work Block?

A Data Manager’s Playbook

Explain the Why

Test Drive the Tools

Clarify Expectations & Define Success

Kick Off and Close Out

A Data Team Member’s Pitch

What do you want to work on?

Why is it important?

Elevator Pitch

Have feedback? I’d love to hear it!

Improving Onboarding for Data Teams

Access / Skills Checklist

Onboarding Journey Doc

Documentation

How will you know it’s working?

A Little Bit of Erika by Your Side²