Sean Lopp
@loppsean.bsky.social
Dagster, RStudio, Looker. Dad. Bikes. Coffee.
Are people calling the Nvidia cosmos paper the next "attention is all you need" but for physical AI systems?
January 8, 2025 at 6:57 PM
Are people calling the Nvidia cosmos paper the next "attention is all you need" but for physical AI systems?
tldr www.youtube.com/watch?v=TRCJ...
End of year we tend to reflect on "Best of...". I've been thinking of _craft excellence_, eg cases where someone produces a masterpiece, an example of them just being excellent at their craft.
End of year we tend to reflect on "Best of...". I've been thinking of _craft excellence_, eg cases where someone produces a masterpiece, an example of them just being excellent at their craft.
Chance the Rapper Performs a Country Version of Nelly’s “Hot in Herre” | That’s My Jam
YouTube video by The Tonight Show Starring Jimmy Fallon
www.youtube.com
December 20, 2024 at 10:53 PM
tldr www.youtube.com/watch?v=TRCJ...
End of year we tend to reflect on "Best of...". I've been thinking of _craft excellence_, eg cases where someone produces a masterpiece, an example of them just being excellent at their craft.
End of year we tend to reflect on "Best of...". I've been thinking of _craft excellence_, eg cases where someone produces a masterpiece, an example of them just being excellent at their craft.
Day 18, happy holidays friends! youtu.be/GjndggmuhCs
30 Days of Orchestration - Day 18
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
December 20, 2024 at 3:39 PM
Day 18, happy holidays friends! youtu.be/GjndggmuhCs
30 Days of Orchestration - Day 17
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
December 19, 2024 at 5:35 PM
30 Days of Orchestration - Day 16
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
December 18, 2024 at 11:08 PM
Day 14 and mayyybbbbee my favorite day so far. Dynamic code gen and peeling back the onion on ways to combine different Python parallelization "layers" .... what more could you want for a Monday? youtu.be/azfCDBKTIoQ
30 Days of Orchestration - Day 14
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
December 16, 2024 at 6:39 PM
Day 14 and mayyybbbbee my favorite day so far. Dynamic code gen and peeling back the onion on ways to combine different Python parallelization "layers" .... what more could you want for a Monday? youtu.be/azfCDBKTIoQ
30 Days of Orchestration - Day 13
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
December 13, 2024 at 5:58 PM
Reposted by Sean Lopp
@loppsean.bsky.social's walk through of asset checks in Day 11 is awesome, do check it out!
In some cases, you might not want to re-read in the asset data (or you might even want to compare across previous materializations!)
In my example, I use Dagster's metadata system to do just that.
In some cases, you might not want to re-read in the asset data (or you might even want to compare across previous materializations!)
In my example, I use Dagster's metadata system to do just that.
![@dg.asset(
deps=[csv_external_asset],
automation_condition=dg.AutomationCondition.on_cron("* * * * *"),
)
def orders(context: dg.AssetExecutionContext, csv_storage: CsvStorageResource) -> None:
orders = csv_storage.read_data('orders_raw.csv')
context.log.info(f"Creating asset one with data: {orders.head()}")
# directly write csv to storage
csv_storage.write_data(orders, 'orders.csv')
return dg.MaterializeResult(
metadata={
"dagster/row_count": dg.MetadataValue.int(len(orders)),
"preview": dg.MetadataValue.md(orders.head().to_markdown()),
"dagster/column_schema": create_table_schema_metadata_from_dataframe(orders)
}
)
@dg.asset_check(asset="orders")
def orders_increasing_or_equal(context: dg.AssetCheckExecutionContext) -> dg.AssetCheckResult:
histotical_metadata = context.instance.fetch_materializations(
dg.AssetRecordsFilter(
asset_key=dg.AssetKey("orders"),
),
limit=2
)
if len(histotical_metadata) < 2:
return dg.AssetCheckResult(passed=True)
row_count_values = [
record.asset_event.metadata.get("dagster/row_count") for record in histotical_metadata[0]
]
latest_row_count = row_count_values[0]
previous_row_count = row_count_values[1]
return dg.AssetCheckResult(
passed=latest_row_count>=previous_row_count,
metadata={"latest_row_count": latest_row_count, "previous_row_count": previous_row_count}
)](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:lc5jzrr425fyah724df3z5ik/bafkreic6g4dx2u76s5odyhy3rnzuszybc3ara4kuvxvud7yo2umvcya33i@jpeg)
December 11, 2024 at 10:22 PM
@loppsean.bsky.social's walk through of asset checks in Day 11 is awesome, do check it out!
In some cases, you might not want to re-read in the asset data (or you might even want to compare across previous materializations!)
In my example, I use Dagster's metadata system to do just that.
In some cases, you might not want to re-read in the asset data (or you might even want to compare across previous materializations!)
In my example, I use Dagster's metadata system to do just that.
Day 11! The diff between orchestration and data orchestration? Your code can stay the same, still work, but the data can change and break everything! youtu.be/ugS1KQhxWrA
30 Days of Orchestration - Day 11
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
December 11, 2024 at 4:46 PM
Day 11! The diff between orchestration and data orchestration? Your code can stay the same, still work, but the data can change and break everything! youtu.be/ugS1KQhxWrA
Reposted by Sean Lopp
For day 10 of 30 days of orchestration, one of my fave topics: materializing assets in different environments using resources!
In my solution to day 10, I added an environment variable, DAGSTER_PROD_DEPLOY, which determines whether to use the folder `data_prod` or `data_dev` if it is not present.
In my solution to day 10, I added an environment variable, DAGSTER_PROD_DEPLOY, which determines whether to use the folder `data_prod` or `data_dev` if it is not present.
![# define a resource that can read and write data
class CsvStorageResource(dg.ConfigurableResource):
base_dir: str = os.path.join(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
"data",
)
def read_data(self, csv_file):
return pd.read_csv(os.path.join(self.base_dir, csv_file) )
def write_data(self, df, csv_file):
return df.to_csv(os.path.join(self.base_dir, csv_file), index=False )
# define an asset that represents an external CSV file
csv_external_asset = dg.AssetSpec(
key = "csv_external_asset",
description="A CSV file that is external to Dagster. If you have issues call 866-5309",
metadata={"file_path": dg.MetadataValue.path("data/orders_raw.csv"),
},
owners=["team:Upstream_Ops", "upstream_ops_person@company.com"],
kinds=["file","csv"],
)
@dg.asset(
deps=[csv_external_asset],
automation_condition=dg.AutomationCondition.on_cron("* * * * *"),
)
def orders(context: dg.AssetExecutionContext, csv_storage: CsvStorageResource) -> None:
orders = csv_storage.read_data('orders_raw.csv')
context.log.info(f"Creating asset one with data: {orders.head()}")
# directly write csv to storage
csv_storage.write_data(orders, 'orders.csv')
return dg.MaterializeResult(
metadata={
"dagster/row_count": dg.MetadataValue.int(len(orders)),
"preview": dg.MetadataValue.md(orders.head().to_markdown()),
"dagster/column_schema": create_table_schema_metadata_from_dataframe(orders)
}
)
@dg.asset(
deps=["orders_summary"],
automation_condition=dg.AutomationCondition.on_cron("*/10 * * * *")
)
def asset_three(context: dg.AssetExecutionContext) -> None:
context.log.info("Creating asset three")
# define the resources to use different folders for different "environments"
resource_defs = {
"DEV":{"csv_storage": CsvStorageResource(base_dir="/Users/christian/code/30_days_of_dagster/data…](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:lc5jzrr425fyah724df3z5ik/bafkreiguofxoye2lcnttae5fc42eclrqicqhzvybxszydzn74oyml33fl4@jpeg)
December 10, 2024 at 9:23 PM
For day 10 of 30 days of orchestration, one of my fave topics: materializing assets in different environments using resources!
In my solution to day 10, I added an environment variable, DAGSTER_PROD_DEPLOY, which determines whether to use the folder `data_prod` or `data_dev` if it is not present.
In my solution to day 10, I added an environment variable, DAGSTER_PROD_DEPLOY, which determines whether to use the folder `data_prod` or `data_dev` if it is not present.
Day 10! Data engineers + Pydantic classes = goodness. youtu.be/f_XfLTlHS-s
30 Days of Orchestration - Day10
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
December 10, 2024 at 5:30 PM
Day 10! Data engineers + Pydantic classes = goodness. youtu.be/f_XfLTlHS-s
Day 9! youtu.be/g4oABNiIOmw
30 Days of Orchestration - Day 9
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
December 9, 2024 at 5:02 PM
Day 9! youtu.be/g4oABNiIOmw
Daty 8, happy Friday! youtu.be/iroAyNM0kxA
30 Days of Orchestration - Day 8
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
December 6, 2024 at 10:01 PM
Daty 8, happy Friday! youtu.be/iroAyNM0kxA
Day 7! youtu.be/u9V0As2RZ5k
30 Days of Orchestration - Day 7
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
December 5, 2024 at 4:46 PM
Day 7! youtu.be/u9V0As2RZ5k
Day 6! youtu.be/RSUk6DU531w
30 Days of Orchestration - Day 6
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
December 4, 2024 at 4:30 PM
Day 6! youtu.be/RSUk6DU531w
Day 5! youtu.be/jNCaAYA-MVU
30 Days of Orchestration - Day 5
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
December 3, 2024 at 4:03 PM
Day 5! youtu.be/jNCaAYA-MVU
Day 4! (Its not too late to catch up!) youtu.be/Jaann2QiGwQ
30 Days of Orchestration - Day 4
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
December 2, 2024 at 4:57 PM
Day 4! (Its not too late to catch up!) youtu.be/Jaann2QiGwQ
Day 3! youtu.be/xUJTal1cq0Y
Learn about more complex DAG scheduling, and get a sneak peek of holiday Daggie
Learn about more complex DAG scheduling, and get a sneak peek of holiday Daggie
30 Days of Orchestration - Day 3
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
November 27, 2024 at 6:19 PM
Day 3! youtu.be/xUJTal1cq0Y
Learn about more complex DAG scheduling, and get a sneak peek of holiday Daggie
Learn about more complex DAG scheduling, and get a sneak peek of holiday Daggie
Reposted by Sean Lopp
Here is my solution: github.com/cnolanminich...
For this one, I wanted to play with the new `target` argument 🤩
For the super common scenario of having one job per schedule, add a "target" argument and select the assets you want -- in my case "asset_one" and everything downstream
#dataBS
For this one, I wanted to play with the new `target` argument 🤩
For the super common scenario of having one job per schedule, add a "target" argument and select the assets you want -- in my case "asset_one" and everything downstream
#dataBS
November 26, 2024 at 9:14 PM
Here is my solution: github.com/cnolanminich...
For this one, I wanted to play with the new `target` argument 🤩
For the super common scenario of having one job per schedule, add a "target" argument and select the assets you want -- in my case "asset_one" and everything downstream
#dataBS
For this one, I wanted to play with the new `target` argument 🤩
For the super common scenario of having one job per schedule, add a "target" argument and select the assets you want -- in my case "asset_one" and everything downstream
#dataBS
Me trying to figure out Power BI pricing
To share content and collaborate in Microsoft Fabric, your organization needs to have an F or P capacity, and at least one per-user license.
To share content and collaborate in Microsoft Fabric, your organization needs to have an F or P capacity, and at least one per-user license.
November 26, 2024 at 7:56 PM
Me trying to figure out Power BI pricing
To share content and collaborate in Microsoft Fabric, your organization needs to have an F or P capacity, and at least one per-user license.
To share content and collaborate in Microsoft Fabric, your organization needs to have an F or P capacity, and at least one per-user license.
Day 2! Its a holiday week, lets talk how you accommodate business schedules with cron
youtu.be/Wja1EmyIexE
youtu.be/Wja1EmyIexE
30 Days of Orchestration - Day 2
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
November 26, 2024 at 5:06 PM
Day 2! Its a holiday week, lets talk how you accommodate business schedules with cron
youtu.be/Wja1EmyIexE
youtu.be/Wja1EmyIexE
I've spent time in all 3 cloud provider consoles, unequivocally Azure is the worst.
I don't mean to offend ... If I am.... DM me. Maybe you can show me the ropes?
Example: the organization > account > subscription hierarchy, when applied to roles, is insane. I am sudo. Respect my authority!
I don't mean to offend ... If I am.... DM me. Maybe you can show me the ropes?
Example: the organization > account > subscription hierarchy, when applied to roles, is insane. I am sudo. Respect my authority!
November 25, 2024 at 9:37 PM
I've spent time in all 3 cloud provider consoles, unequivocally Azure is the worst.
I don't mean to offend ... If I am.... DM me. Maybe you can show me the ropes?
Example: the organization > account > subscription hierarchy, when applied to roles, is insane. I am sudo. Respect my authority!
I don't mean to offend ... If I am.... DM me. Maybe you can show me the ropes?
Example: the organization > account > subscription hierarchy, when applied to roles, is insane. I am sudo. Respect my authority!
Lets GO! Day 1 of 30 days of orchestration
youtu.be/RP6B1wXug_0
This will certainly be the most contentious video, because it starts by creating a Python environment (and of course, includes debugging said environment live).
youtu.be/RP6B1wXug_0
This will certainly be the most contentious video, because it starts by creating a Python environment (and of course, includes debugging said environment live).
30 Days of Orchestration - Day 1
YouTube video by Sean Lopp @ Dagster Labs
youtu.be
November 25, 2024 at 2:43 PM
Lets GO! Day 1 of 30 days of orchestration
youtu.be/RP6B1wXug_0
This will certainly be the most contentious video, because it starts by creating a Python environment (and of course, includes debugging said environment live).
youtu.be/RP6B1wXug_0
This will certainly be the most contentious video, because it starts by creating a Python environment (and of course, includes debugging said environment live).
There are two types of Git commit messages:
8:00am: ae678pt: "as described in the linked RFC, implementing changes to the partition strategy to reduce hotspots in our cluster index"
4:49pm: stcm98: "f-ing typos, try again. pleeeeease pass"
8:00am: ae678pt: "as described in the linked RFC, implementing changes to the partition strategy to reduce hotspots in our cluster index"
4:49pm: stcm98: "f-ing typos, try again. pleeeeease pass"
November 20, 2024 at 10:34 PM
There are two types of Git commit messages:
8:00am: ae678pt: "as described in the linked RFC, implementing changes to the partition strategy to reduce hotspots in our cluster index"
4:49pm: stcm98: "f-ing typos, try again. pleeeeease pass"
8:00am: ae678pt: "as described in the linked RFC, implementing changes to the partition strategy to reduce hotspots in our cluster index"
4:49pm: stcm98: "f-ing typos, try again. pleeeeease pass"