alex? alex!
LightNews LightNews
elx.bsky.social
alex? alex!
@elx.bsky.social
36 followers 74 following 30 posts
Your main character of the day, everyday.
Posts Replies Media Videos
elx.bsky.social
After working purely in Go for the past 2 years, I have really been loving using @deno.land for side projects and scripts. Since it follows so many existing ES standards, I can basically guess how to do most things. Streaming a large CSV and getting unique URLs is as simple as the below.
// Deno Code for Parsing a file import { CsvParseStream } from '@std/csv'; class DistinctUrlStream extends TransformStream { seenUrls = new Set<string>() constructor() { super({ transform: (chunk: Record<string, string>, controller) => { const { urls } = chunk; for (const url of urls.split(', ')) { if (this.seenUrls.has(url)) { continue; } this.seenUrls.add(url); controller.enqueue(`${url}\n`); } }}); } } const output = await Deno.open('./unique-output.txt', { create: true, append: true, }); const input = await Deno.open('./input.csv'); input.readable .pipeThrough(new TextDecoderStream()) .pipeThrough(new CsvParseStream({ trimLeadingSpace: true, skipFirstRow: true })) .pipeThrough(new DistinctUrlStream()) .pipeThrough(new TextEncoderStream()) .pipeTo(output.writable);
November 23, 2024 at 8:19 PM