# Novatise datasets, release 2026-11-12

Built from the corpus export of 2026-11-11 (20,232 published posts: every WordPress site on the box plus every Plikli tenant). Built by `docs/corpus-review-2026-11-11/data/scripts/build_release.py` in the server-build repo; rerun it to rebuild. Counts are in `stats.json`; file checksums in `SHA256SUMS`.

Files are CSV, UTF-8 with BOM (opens correctly in Excel, Japanese included). Every row carries the URL of the post it came from.

## What is in it, and how far to trust it

| File | Rows | What it is | Trust |
|---|---|---|---|
| `1_demand_products.csv` | 3,511 products | Products the network has a "where to buy" post for, with how many posts and sites ask about each, and which stores those posts cover | High as "what people ask about". Says nothing about real availability |
| `1_product_strings.csv` | 9,538 (9,442 distinct, 1,268 ASINs) | Product strings already attached to posts: PriceBest box queries, Pochipp affiliate titles, Amazon ASINs | High as identifiers. Pochipp titles are shop listing titles: use for matching, never display |
| `2_sg_menu_prices.csv` | 22,567 items, 219 restaurants | Singapore menu items with price in SGD (menuprices.sg) | Extraction accurate. Prices are undated and partly stale; see year columns |
| `3_cosmetic_ingredients.csv` | 1,375 rows, 458 ingredients, 92 posts | Ingredient → cosmetic function (restorbio.com, haireveryday.com) | Looks correct. Source uncited; likely copied from a public decoder site |
| `4_snack_nutrition.csv` | 177 products | Energy, protein, fat, carbohydrate, salt, plus contents / release date / ingredients where the post's table had them | Values look copied from maker pages; no source, no check date |
| `5_faq_pairs.csv` | 12,974 pairs, 57 sites | Question–answer pairs from explicit FAQ sections | Answers inherit the posts' weaknesses; flags below |

## Deliberately left out

- **hugemagazine.jp**: removed from every file. Cosmetic-surgery allegations about named real people; a liability, not a dataset.
- **afterhoursmagazine.jp** from the FAQ file: gossip about real people.
- **Store availability claims** (30,391 in the raw extract). Not a single one is verified: 46% are hedged in their own words, 36% bare assertions, 14% rest on a tweet. Only the fact that a post *asks* about a product is kept (file 1). The raw claims stay in `..\derived\datasets\product_store_availability.csv` as a lead list for verification.
- **Discontinuation claims, tweets, celebrity and idol profiles, AliExpress/DHgate links, encyclopedia prose**: available raw in `..\derived\datasets\`, not cleaned for release. See `docs/corpus-review-2026-11-11/data/DATASET-OPPORTUNITIES.md` for why.

## Columns that need care

**1_demand_products.csv**
- `posts`, `sites`: how many posts / sites ask about the product. Ranked by these.
- `stores_covered`: stores the posts *discuss*, not stores that stock it. Names are normalised; `(総称)` means a generic category (all convenience stores, all online shops).
- Product names come from post titles: about 91% correct on a hand check before clean-up. Some are long fragments ("100均で教科書クリップ").

**2_sg_menu_prices.csv**
- `year_stated`: year printed in the table heading, else in the post title. `year_source` says which.
- `year_source = none`: 5,987 items carry no year at all.
- `year_conflict = 1`: 2,433 items sit under a table whose year differs from the post title (e.g. a "2025" post showing "2023 Prices").
- 4,382 items carry a year of 2022 or older.
- `plus_plus = 1`: price written with "++" (service charge and GST extra). Everything else: basis unknown.
- No outlet, no source, no capture date. Treat as historical snapshots, not current prices.

**3_cosmetic_ingredients.csv**
- `ingredient_key`: lower-cased name for matching ("Disodium Edta" = "Disodium EDTA"). Not full INCI normalisation.
- `function_claimed`: as written in the post. For publication, rebuild functions from the EU CosIng database rather than reuse these.

**4_snack_nutrition.csv**
- Values as printed, units included (`57kcal`, `1.1g`). Formulas change; confirm against the maker before reuse.

**5_faq_pairs.csv**
- `answer_hedged = 1`: answer says 「ようです」「みたいです」 etc. (112 rows).
- `answer_first_person = 1`: answer claims personal use ("私の…", "使ってみ…"); these are persona claims and unverifiable (187 rows).
- Short and shortcode-only answers removed; exact repeats removed.

## Known limits

- Nothing here was checked against stores, manufacturers or the live web.
- Accuracy figures come from the review's hand-checked samples (see `docs/corpus-review-2026-11-11/data/EXTRACTION-REPORT.md` and its `validation\` folder). Clean-up applied here after those checks was not re-measured.
- ASINs sit on 142 fishing and gear posts, not on the where-to-buy posts, so they do not join to file 1.

## Verification pass (2026-11-12) and new sourced datasets

The checks and their results are in `verification\` (one `*-CHECK.md` per check, plus every script used):
- **Ingredients:** all 1,375 rows checked against EU CosIng. 81.6% of rows have every claimed function listed by CosIng. The posts mention EU-prohibited (Annex II) hair dyes and zinc pyrithione with no warning. See `INGREDIENTS-CHECK.md`.
- **Snack nutrition:** 79 of 177 rows checked against the maker's own page. 25 match the current label, 25 differ (mostly older labels or a different serving), and 29 Bourbon products are gone from Bourbon's catalogue. 98 not checked. See `NUTRITION-CHECK.md`.
- **SG menus:** 2,029 rows from six chains checked against the current official menus; 61% are still on the menu. Prices could not be checked (no chain shows prices without picking a store). See `MENU-CHECK.md`.

New files, built only from those official sources:

| File | Rows | What it is |
|---|---|---|
| `6_ingredient_glossary_cosing.csv` | 379 | INCI name, EU CosIng functions, Annex entry and a prohibited flag, the names the posts use, and how many posts mention each. Source URL on every row |
| `7_discontinuation_vs_maker.csv` | 101 | "Is X discontinued?" posts about Bourbon, Glico, Meiji and Lotte products, against whether the maker still lists the product (2026-11-12) |
| `8_snack_nutrition_maker_sourced.csv` | 50 | Nutrition taken from the maker's page (not the post), with the source URL, check date, and whether the post's figures agreed |

### 9_maker_nutrition_catalogue.csv (added 2026-11-12)
1,887 products with nutrition labels, read from the makers' own product pages on 2026-11-12: Meiji 867, Glico 449, Lotte 347 and Bourbon 224. Columns: maker, product, per (the serving the label is for), kcal, protein_g, fat_g, carb_g, salt_g, source_url, checked. The values were parsed from page text. 345 rows have no product name, mostly Lotte, whose pages carry no title; the source_url identifies them. Pages listing several products were dropped. Built by verification/build_maker_catalogue.py.
(9 was rebuilt later on 2026-11-12: Morinaga added from its own product pages and blank names filled, giving 2,114 rows.)

### 10_product_ingredients.csv
1,638 ingredient lists (原材料名) from the makers' product pages, read on 2026-11-12. Columns: maker, product, ingredients, source_url, checked. Pages where the ingredient text could not be read cleanly are left out.

### 11_product_allergens.csv
1,169 products, Bourbon not included, with the allergens listed on the maker's page. Only the 28 Japanese labelling items count (8 mandatory and 20 recommended). The basis column says where each list came from:
- "allergen line": the allergen section of the page.
- "allergen grid": Morinaga's ● marks. Only the 8 mandatory items were in the captured text.
- "ingredients note": the (一部に…を含む) note in the ingredients.

A product with no allergen named on the page gets no row; nothing is recorded as allergen-free. Bourbon's allergen block could not be parsed reliably, so Bourbon is left out. This is not allergy advice: check the pack.

### PriceBest product-name check
800 PriceBest watch items were compared with the corpus product names (verification/pricebest_match.csv). Only about 16 matched, too few to verify names with this source.