will/osm-import-tools
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Create new project design plans

Browse Source
This commit is contained in:
zyphlar
2026-08-28 14:23:26 -07:00
parent a7d6db8177
commit 7ce73f19c5
22 changed files with 2682 additions and 263 deletions
Download Patch File Download Diff File Expand all files Collapse all files
map-checker/README.md
+70
View File
@@ -0,0 +1,70 @@
# map-checker
Cross-references our Sumter/Lake county address data against a reference map
(e.g. [thevillages.com/map](https://www.thevillages.com/map/)) to catch city
boundary or misnaming issues (e.g. addresses that are really in The Villages
but our data has as Lady Lake, Sumterville, Leesburg, etc.).
## Why this is manual, not automated
thevillages.com's Terms of Use explicitly prohibit automated access:
> "You must not use any robot, spider or other automatic device, process or
> means to access the Website for any purpose, including monitoring or
> copying any of the material on the Website."
So this subproject never fetches, scrapes, or otherwise programmatically
touches their site. Every input here comes from a human looking at the
official map in their own browser and either tracing a boundary or recording
an observation by hand.
## Workflow 1: hand-traced boundary (recommended)
Recording individual addresses one at a time doesn't scale to tens of
thousands of records. Instead, trace the boundary **once** and check every
address against it in one pass.
1. Open `https://www.thevillages.com/map/` (or another reference map) in one
browser tab.
2. Open **`/map-checker`** in the app (the "Boundary Tracer" link on the main
page) in another tab.
3. Click "New polygon", then click the map to place vertices tracing the
boundary you see on the reference map. The Villages is disjoint across
three counties, so you'll likely want several separate polygons - just
name them all the same (e.g. "The Villages") and add another with "New
polygon" once you finish the current one.
4. Click "Finish polygon", give it a name, repeat for each disjoint piece.
5. Click "Save to server" (requires login) - this writes
`/data/map-checker/boundary.geojson`. "Download GeoJSON" also works if you
just want a local copy.
6. Run the point-in-polygon check:
```bash
docker compose exec web python3 map-checker/check-boundary.py --county lake --county sumter
```
This flags:
- addresses **inside** the traced boundary whose county/OSM city fields
don't say "The Villages" (probably under-tagged), and
- addresses **outside** it that do say "The Villages" anyway (probably
over-tagged or the boundary needs another pass).
Pass `--expect-city "Some Other City"` to check a different boundary/name,
and `--output flagged.geojson` to save the flagged points for review in
the main map viewer.
## Workflow 2: individual spot-checks
For one-off verification of a specific address (not a bulk boundary check),
[`reference.yml`](reference.yml) + `check.py` still works the same way as
before: hand-record what the reference map says for a specific street/area,
then run `python check.py` to compare it against our data. See the comments
at the top of `reference.yml` for the entry format.
## Scope
`check-boundary.py` is only as accurate as the hand-traced polygon - it's a
spot-check/audit tool, not a source of truth. For a systematic (but coarser)
look at where the county's own postal-community and city/jurisdiction fields
disagree with each other and with OSM, see the "City Boundaries" report in
the main app (`/reports/city-boundaries`).
map-checker/check-boundary.py
+163
View File
@@ -0,0 +1,163 @@
#!/usr/bin/env python3
"""
Point-in-polygon check: for a hand-traced boundary (see the Boundary Tracer
tool at /map-checker, or draw manually and save as boundary.geojson),
flag every county/OSM address that disagrees with it - either inside the
boundary but not labeled with the expected city, or outside it but labeled
with the expected city anyway.
The boundary itself is drawn by a human looking at a reference map in their
own browser (e.g. thevillages.com/map) - nothing here fetches from any
third-party site, only our own /data/{county}/*.geojson files.
Usage:
python check-boundary.py --county lake --county sumter
python check-boundary.py --county lake --expect-city "The Villages" --output flagged.geojson
"""
import argparse
import json
import sys
from pathlib import Path
from shapely.geometry import shape, Point
from shapely.ops import unary_union
import warnings
warnings.filterwarnings('ignore')
def normalize_city(value):
if value is None:
return ''
s = str(value).strip()
return '' if s in ('', 'nan', 'None') else s
def load_boundary(path):
with open(path) as f:
data = json.load(f)
polys = []
for feat in data.get('features', []):
name = (feat.get('properties') or {}).get('name', 'boundary')
polys.append((name, shape(feat['geometry'])))
return polys
def load_addresses(data_dir, county):
county_dir = Path(data_dir) / county
# Prefer city-comparison.geojson: has postal community, city, AND osm_city
# together for matched addresses. Falls back to county-addresses.geojson
# (county-only, addr:city, no OSM side) if the diff step hasn't run yet.
for filename in ('city-comparison.geojson', 'county-addresses.geojson'):
path = county_dir / filename
if path.exists():
with open(path) as f:
return json.load(f), filename
return None, None
def city_values(props):
values = [
normalize_city(props.get('county_postal_community')),
normalize_city(props.get('county_city')),
normalize_city(props.get('osm_city')),
normalize_city(props.get('addr:city')),
]
return [v for v in values if v]
def main():
parser = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument('--boundary', default='/data/map-checker/boundary.geojson',
help='Path to the traced boundary GeoJSON (default: the file saved by '
'the Boundary Tracer web tool at /map-checker)')
parser.add_argument('--data-dir', default='/data')
parser.add_argument('--county', action='append',
help='County id to check (repeatable). Default: lake, sumter')
parser.add_argument('--expect-city', default='The Villages',
help='City name addresses inside the boundary should have (default: "The Villages")')
parser.add_argument('--output', help='Optional path to save flagged addresses as GeoJSON')
args = parser.parse_args()
boundary_path = Path(args.boundary)
if not boundary_path.exists():
print(f"Error: {boundary_path} not found.", file=sys.stderr)
print("Trace a boundary first via the Boundary Tracer tool (/map-checker), "
"or save one to that path by hand.", file=sys.stderr)
return 1
polys = load_boundary(boundary_path)
if not polys:
print("Error: boundary file has no polygons.", file=sys.stderr)
return 1
combined = unary_union([g for _, g in polys])
counties = args.county or ['lake', 'sumter']
expect = args.expect_city.strip().lower()
total = inside_count = inside_flagged = outside_flagged = 0
flagged_features = []
print(f"Boundary: {', '.join(sorted(set(name for name, _ in polys)))} "
f"({len(polys)} polygon(s))")
for county in counties:
fc, source_file = load_addresses(args.data_dir, county)
if fc is None:
print(f" {county}: no data found, skipping.")
continue
county_total = county_inside = county_inside_flagged = county_outside_flagged = 0
for feat in fc['features']:
coords = feat['geometry']['coordinates']
pt = Point(coords[0], coords[1])
is_inside = combined.contains(pt)
props = feat.get('properties', {})
values = city_values(props)
says_expected = any(v.lower() == expect for v in values)
county_total += 1
if is_inside:
county_inside += 1
if values and not says_expected:
county_inside_flagged += 1
flagged_features.append({
**feat,
'properties': {**props, 'county': county, 'flag': 'inside_but_not_labeled'},
})
elif says_expected:
county_outside_flagged += 1
flagged_features.append({
**feat,
'properties': {**props, 'county': county, 'flag': 'outside_but_labeled'},
})
print(f" {county} ({source_file}): {county_total} addresses, "
f"{county_inside} inside boundary "
f"({county_inside_flagged} not labeled '{args.expect_city}'), "
f"{county_outside_flagged} outside boundary but labeled '{args.expect_city}'")
total += county_total
inside_count += county_inside
inside_flagged += county_inside_flagged
outside_flagged += county_outside_flagged
print(f"\n{'=' * 60}")
print(f"Total: {total} addresses checked, {inside_count} inside the boundary.")
print(f" Inside but not labeled '{args.expect_city}': {inside_flagged}")
print(f" Outside but labeled '{args.expect_city}': {outside_flagged}")
if args.output:
out = {'type': 'FeatureCollection', 'features': flagged_features}
with open(args.output, 'w') as f:
json.dump(out, f)
print(f"\nSaved {len(flagged_features)} flagged addresses to {args.output}")
return 0
if __name__ == '__main__':
sys.exit(main())
map-checker/check.py
+209
View File
@@ -0,0 +1,209 @@
#!/usr/bin/env python3
"""
map-checker: compare hand-recorded observations from thevillages.com/map
(see reference.yml) against our county/OSM address data.
This is a deliberately manual workflow. thevillages.com's Terms of Use
prohibit automated/bot access to their site, so nothing here fetches from
them - reference.yml is filled in BY HAND after a human looks at the
official map in their own browser. This script only reads OUR OWN data
(county-addresses.geojson / city-comparison.geojson, already produced by
convert-addresses.py and generate-city-report.py) and checks it against
those manually recorded entries.
Usage:
python check.py [--reference reference.yml] [--data-dir /data] [--radius 200]
"""
import argparse
import json
import math
import re
import sys
from pathlib import Path
import yaml
DEFAULT_RADIUS_METERS = 200.0
def normalize_city(value):
if not value:
return ''
s = str(value).strip()
return '' if s in ('', 'nan', 'None') else s
def normalize_street(value):
if not value:
return ''
s = str(value).strip().lower()
s = re.sub(r'\bstate road\b', 'sr', s)
s = re.sub(r'\bstate route\b', 'sr', s)
s = re.sub(r'\bcounty road\b', 'cr', s)
s = re.sub(r'\s+', ' ', s).strip()
return s
def haversine_meters(lat1, lon1, lat2, lon2):
r = 6371000.0
p1, p2 = math.radians(lat1), math.radians(lat2)
dp = math.radians(lat2 - lat1)
dl = math.radians(lon2 - lon1)
a = math.sin(dp / 2) ** 2 + math.cos(p1) * math.cos(p2) * math.sin(dl / 2) ** 2
return 2 * r * math.asin(math.sqrt(a))
def load_geojson(path):
if not path.exists():
return None
with open(path) as f:
return json.load(f)
def find_matches(entry, county_features, comparison_features, radius_m):
"""Return a list of matched feature-property dicts for one reference entry.
Prefers city-comparison.geojson (has county_postal_community, county_city,
AND osm_city together) when available; falls back to the raw
county-addresses.geojson (county-only, no OSM side) otherwise.
"""
source = comparison_features if comparison_features is not None else county_features
if source is None:
return []
street = entry.get('street')
housenumber = entry.get('housenumber')
lat, lon = entry.get('lat'), entry.get('lon')
matches = []
for feature in source['features']:
props = feature.get('properties', {})
if lat is not None and lon is not None:
coords = feature['geometry']['coordinates']
dist = haversine_meters(lat, lon, coords[1], coords[0])
if dist > radius_m:
continue
elif street:
if normalize_street(props.get('addr:street')) != normalize_street(street):
continue
if housenumber is not None:
try:
if int(props.get('addr:housenumber')) != int(housenumber):
continue
except (TypeError, ValueError):
continue
else:
continue
matches.append(props)
return matches
def check_entry(entry, index, county_features, comparison_features, radius_m):
label = entry.get('street') or f"({entry.get('lat')}, {entry.get('lon')})"
official_city = normalize_city(entry.get('official_city'))
matches = find_matches(entry, county_features, comparison_features, radius_m)
print(f"\n[{index}] {label}"
+ (f" #{entry['housenumber']}" if entry.get('housenumber') else '')
+ f" ({entry.get('county', '?')})")
if entry.get('official_village'):
print(f" Official village: {entry['official_village']}")
print(f" Official city (recorded): {official_city or '(none recorded)'}")
if not matches:
print(" No matching addresses found in our data - can't check.")
return {'checked': True, 'matched': False, 'mismatches': 0}
# A single street address (e.g. a shopping plaza or apartment building)
# can match many unit rows that all carry identical city values - group
# those together instead of printing one line per unit.
groups = {} # (postal_community, city, osm_city) -> {'count', 'addr'}
for props in matches:
addr = ' '.join(str(v) for v in
[props.get('addr:housenumber'), props.get('addr:street')] if v)
key = (normalize_city(props.get('county_postal_community')),
normalize_city(props.get('county_city')),
normalize_city(props.get('osm_city')))
if key not in groups:
groups[key] = {'count': 0, 'addr': addr}
groups[key]['count'] += 1
mismatches = 0
for (postal, city, osm), g in groups.items():
fields = [('postal community', postal), ('city', city), ('OSM city', osm)]
parts = []
row_mismatch = False
for field_label, value in fields:
if not value:
continue
agrees = (not official_city) or value.lower() == official_city.lower()
if not agrees:
row_mismatch = True
flag = '' if agrees else ' <-- MISMATCH'
parts.append(f"{field_label}={value}{flag}")
count_note = f" (x{g['count']})" if g['count'] > 1 else ''
if row_mismatch:
mismatches += g['count']
print(f" {g['addr']}{count_note}: " + ', '.join(parts))
return {'checked': True, 'matched': True, 'mismatches': mismatches}
def main():
parser = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument('--reference', default=str(Path(__file__).parent / 'reference.yml'))
parser.add_argument('--data-dir', default='/data')
parser.add_argument('--radius', type=float, default=DEFAULT_RADIUS_METERS,
help=f'Match radius in meters for lat/lon entries (default {DEFAULT_RADIUS_METERS})')
args = parser.parse_args()
ref_path = Path(args.reference)
if not ref_path.exists():
print(f"Error: {ref_path} not found.", file=sys.stderr)
return 1
with open(ref_path) as f:
data = yaml.safe_load(f) or {}
checks = data.get('checks') or []
if not checks:
print(f"No entries in {ref_path} yet.")
print("Open https://www.thevillages.com/map/ in your own browser, look up an address,")
print("and add an entry to reference.yml by hand - see the comments at the top of that file.")
return 0
data_dir = Path(args.data_dir)
cache = {}
total_mismatches = 0
total_unmatched = 0
for i, entry in enumerate(checks, 1):
county = entry.get('county')
if not county:
print(f"\n[{i}] Skipped - missing 'county' field.")
continue
if county not in cache:
cache[county] = (
load_geojson(data_dir / county / 'county-addresses.geojson'),
load_geojson(data_dir / county / 'city-comparison.geojson'),
)
county_features, comparison_features = cache[county]
result = check_entry(entry, i, county_features, comparison_features, args.radius)
if not result['matched']:
total_unmatched += 1
total_mismatches += result['mismatches']
print(f"\n{'=' * 60}")
print(f"Checked {len(checks)} entries: {total_mismatches} address(es) with a "
f"disagreeing city field, {total_unmatched} entries with no match in our data.")
return 0
if __name__ == '__main__':
sys.exit(main())
map-checker/reference.yml
+32
View File
@@ -0,0 +1,32 @@
# map-checker reference log
#
# This file is filled in BY HAND. Open https://www.thevillages.com/map/ in
# your own browser, look up a street/area, and record what the official map
# says here. Nothing in this subproject fetches, scrapes, or otherwise
# automates access to thevillages.com — their Terms of Use explicitly
# prohibit automated/bot access ("You must not use any robot, spider or
# other automatic device, process or means to access the Website for any
# purpose, including monitoring or copying any of the material on the
# Website"), so every entry here is a manual, human-recorded observation.
#
# Fields:
# street - street name as it appears in county/OSM data (required
# unless lat/lon given instead)
# housenumber - optional, narrows the match to one address
# lat / lon - optional alternative to street: a point you looked up
# on the official map (e.g. right-clicked or geocoded)
# county - which county's data to check against: sumter | lake
# official_village - the named Village shown on thevillages.com/map (e.g.
# "Fenney", "DeLuna"), if shown - informational only
# official_city - what city/community the official map (or other
# authoritative source you consulted) indicates this
# address belongs to
# recorded_by - your name/username
# recorded_date - YYYY-MM-DD
# notes - anything else worth remembering (screenshots taken,
# ambiguity noticed, source other than the map, etc.)
#
# Run `python check.py` after adding entries to compare them against our
# county/OSM data and flag disagreements.
checks: []