will/osm-import-tools
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Improve address matching

Browse Source
This commit is contained in:
zyphlar
2026-02-25 10:25:49 -08:00
parent ea3e4ca098
commit bdaafd85ef
3 changed files with 168 additions and 289 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DOCKER.md
-117
View File
@@ -1,117 +0,0 @@
# Docker Deployment Guide
## Quick Start
### Using Docker Compose (Recommended)
1. Build and start the container:
```bash
docker-compose up -d --build
```
Note: The `--build` flag ensures the image is rebuilt with the latest code changes.
2. Access the web interface:
- Main interface: http://localhost:5000
- Map viewer: http://localhost:5000/map
3. Stop the container:
```bash
docker-compose down
```
### Using Docker Directly
1. Build the image:
```bash
docker build -t villages-import .
```
2. Run the container:
```bash
docker run -d \
-p 5000:5000 \
-v "$(pwd)/data:/data" \
--name villages-import \
villages-import
```
3. View logs:
```bash
docker logs -f villages-import
```
4. Stop the container:
```bash
docker stop villages-import
docker rm villages-import
```
## Features
### Main Dashboard (/)
- Run data processing scripts for Lake and Sumter counties
- View real-time script output
- Access to:
- Diff Roads
- Diff Addresses
- Diff Multi-Use Paths
- Download OSM Data
### Map Viewer (/map)
- Interactive map viewer for GeoJSON files
- Upload and compare OSM, Diff, and County data
- Filter by removed/added features
- Hide highway=service roads
- Drag-and-drop layer reordering
- Click on features to view properties
- Accept/reject diff features
## Volume Mounts
The Docker container mounts a single data directory:
- `./data` → `/data` - All data files, organized by date
Inside `/data`, the structure is:
- `/data/latest/` - Symlink to the most recent data directory
- `/data/YYMMDD/lake/` - Lake County data for that date
- `/data/YYMMDD/sumter/` - Sumter County data for that date
All changes are persisted on the host in the local `./data` folder.
## API Endpoints
- `GET /` - Main dashboard
- `GET /map` - Map viewer
- `POST /api/run-script` - Execute a processing script
- `GET /api/job-status/<job_id>` - Get script status and logs
- `GET /api/list-files` - List available GeoJSON files
- `GET /data/<path>` - Serve GeoJSON files
## Troubleshooting
### Port already in use
If port 5000 is already in use, edit `docker-compose.yml`:
```yaml
ports:
- "8080:5000" # Change 8080 to any available port
```
### Permission issues
Ensure the data directory has proper permissions:
```bash
mkdir -p data
chmod -R 755 data
```
### View container logs
```bash
docker-compose logs -f
```
### Rebuild after code changes
```bash
docker-compose down
docker-compose build --no-cache
docker-compose up -d
```
README.md
+113 -158
View File
@@ -1,169 +1,124 @@
# The Villages Road and Address Import # OSM Import Tools
See [https://wiki.openstreetmap.org/wiki/The_Villages_Road_and_Address_Import](https://wiki.openstreetmap.org/wiki/The_Villages_Road_and_Address_Import) ## Docker Quick Start
See compare-addresses.py for an automated way of running the complete address diff toolchain in one step. ### Using Docker Compose (Recommended)
- TODO: fails to split out units
## New Instructions 1. Build and start the container:
```bash
* NOTE: when downloading OSM data towards the end via JOSM, copy-paste the output of the download script but add `(._;>;);out;` to the end instead of `out geom;` so JOSM picks it up. docker-compose up -d --build
* NOTE: also add `way["highway"="construction"](area.searchArea);way["highway"="path"](area.searchArea);way["highway"="cycleway"](area.searchArea);` to the end so that roads under construction and cartpaths show up in JOSM to be analyzed/replaced/modified/etc.
### Roads
* Get new data from the county and convert it:
* Sumter (change 041125): `python shp-to-geojson.py "original data/Sumter/RoadCenterlines_041125.shp.zip" "original data/Sumter/RoadCenterlines_041125.geojson"`
* Lake (change 2025-06): `python shp-to-geojson.py "original data/Lake/Streets 2025-06.zip" "original data/Lake/Streets 2025-06.geojson"`
* Get new data from OSM:
* Sumter: `python download-overpass.py --type highways "Sumter County" "Florida" "original data/Sumter/osm-sumter-roads-$(date +%y%m%d).geojson"`
* Lake: `python download-overpass.py --type highways "Lake County" "Florida" "original data/Lake/osm-lake-roads-$(date +%y%m%d).geojson"`
* Diff the roads:
* Sumter (change 041125): `python threaded.py --output "processed data\Sumter\diff-sumter-roads-$(date +%y%m%d).geojson" "original data\Sumter\osm-sumter-roads-$(date +%y%m%d).geojson" "original data\Sumter\RoadCenterlines_041125.geojson"`
* Lake (change 2025-06): `python threaded.py --output "processed data\Lake\diff-lake-roads-$(date +%y%m%d).geojson" "original data\Lake\osm-lake-roads-$(date +%y%m%d).geojson" "original data\Lake\Streets 2025-06.geojson"`
## Data
- Lake County Streets and Address Points: https://c.lakecountyfl.gov/ftp/GIS/GisDownloads/Shapefiles/
- Alternately:
- Streets: https://gis.lakecountyfl.gov/lakegis/rest/services/InteractiveMap/MapServer/73
- Addresses: https://gis.lakecountyfl.gov/lakegis/rest/services/InteractiveMap/MapServer/16
- Highways: https://gis.lakecountyfl.gov/lakegis/rest/services/InteractiveMap/MapServer/9
- Sumter GIS:
- Alternately, roads: https://test-sumter-county-open-data-sumtercountygis.hub.arcgis.com/datasets/9177e17c72d3433aa79630c7eda84add/about
- Addresses: https://test-sumter-county-open-data-sumtercountygis.hub.arcgis.com/datasets/c75c5aac13a648968c5596b0665be28b/about
- Email for Multi-Modal Paths.
- Marion (TODO)
## Instructions
* Always do roads first, addresses second, so new subdivisions don't throw address validation errors.
* Open the original data in QGIS
* Format OSM fields with QGIS functions to have proper capitalization and full spellings without extraneous whitespace, based on original fields. For example OSM uses names like North Main Street, not N MAIN ST. All fields are of the QGIS type "text" even if they're numbers.
* You can use the Attribute Table's Field Calculator for this; you can copy-paste the `qgis-functions.py` file into the Function Editor and then use the Expression tab to create new, formatted virtual fields. Don't worry if the field name limit is too short, it can be fixed in JOSM.
### For Sumter County:
* Always use the Filter with Form function to Select all entries with `"LIFECYCLE"='Current'`
* For roads:
* `NAME` becomes the virtual `name` via the `title(formatstreet("NAME"))`
* `SpeedLimit` becomes the virtual `maxspeed` via `concat("SpeedLimit",' mph')`
* `highway=residential` or similar added manually in JOSM
* `surface=asphalt` added manually in JOSM
* For addresses:
* The Addresses shapefile is recorded in the ESRI:102659 CRS, you may need to convert or reproject to/from the default EPSG:4326 - WGS 84 CRS that OSM uses.
* `ADD_NUM` becomes the virtual `addr:housenumber` (or `addr:house` temporarily, avoiding addr:house which is a real tag) as an integer
* `UNIT` becomes the virtual `addr:unit` (sometimes the LOT key is used for multiple units in a range, but mostly it's unrelated lot IDs and not useful) as a string
* `SADD` becomes the virtual `addr:street` (or `addr:stree` temporarily) via the `title(getstreetfromaddress("SADD"))` custom expression as a string
* `POST_COMM` becomes the virual `addr:city` via the `title("POST_COMM")` expression (we care about postal community addresses not what municipality a place might be governed by) as a string
* `POST_CODE` becomes `addr:postcode` (or `addr:postc` temporarily) as an integer
* Manually add `addr:state` = `'FL'`
* For multi-modal trails (golf cart paths):
* Download all highway=path and highway=cycleway with golf_cart=yes for comparison
* Omit `Part_of_Ro`=`Yes` as separate paths; apply golf cart tagging to the streets directly.
* `bicycle=yes`
* `foot=yes`
* `golf=cartpath`
* `golf_cart=yes`
* `highway=path`
* `motor_vehicle=no`
* `segregated=no`
* `surface=asphalt`
### For Lake County:
* For roads:
* `FullStreet` becomes the virtual `name` via the `title(formatstreet("FullStreet"))`
* `SpeedLimit` becomes the virtual `maxspeed` via `concat("SpeedLimit",' mph')`
* `NumberOfLa` becomes the virtual `lanes`
* `surface=asphalt` added manually
* `StreetClas` becomes the virtual `highway` via the `gethighwaytype("StreetClas")`
* Could use MaxWeight (1.0 - 20.0)
* For addresses:
* The Addresses shapefile is recorded in the NAD83(HARN) / Florida East (ftUS) CRS, you may need to convert or reproject to/from the default EPSG:4326 - WGS 84 CRS that OSM uses.
* `AddressNum` becomes the virtual `addr:housenumber` (or `addr:house` temporarily, avoiding addr:house which is a real tag) as an integer
* `UnitType` becomes the virtual `addr:unit` via `regexp_replace("UnitType",'U ','')` (UnitNumber is blank) as a string
* The virtual `addr:street` (or `addr:stree` temporarily) is created via the `regexp_replace(trim(concat(formatname("PrefixDire"),' ',title(formatstreet("PrefixType")),' ',title(formatstreet("BaseStreet")),' ',formatname("SuffixType"))),'\\s+',' ')` custom expression as a string
* `PostalCity` becomes the virual `addr:city` via the `title("PostalCity")` expression (we care about postal community addresses not what municipality a place might be governed by) as a string
* `ZipCode` becomes `addr:postcode` (or `addr:postc` temporarily) as an integer
* Manually add `addr:state` = `'FL'`
### Continuing instructions for both:
* Export to Geojson, only exporting **selected** entries, **selecting only the OSM-formatted fields we want**.
* Here you can rename temporary columns like `addr:house` to `addr:housenumber`.
* Ensure the export file is in the `EPSG:4326 - WGS84` CRS.
* Open in JOSM. It's suggested to begin with roads first, addresses second, so the addresses can be placed in context.
* In the Roads dataset, select and remove all relations from the geojson/shapefile layer: the data often has one relation per road and this is improper for OSM import.
* Select a small region to work on: one neighborhood or smaller. For this import, we are assuming that only newly-constructed small residential areas will be imported, not main roads or commercial areas or areas with significant existing map data.
* Download the area you're working on from OSM, into a new Data Layer (not your geojson layer.)
* Select all features to be imported at this time and leave them selected until the merge step below.
* Select all ways for roads, or all nodes for addresses. Make sure you aren't about to mass-edit the nodes of a road: deselect the nodes if this happens.
* Ensure the tags are correct and good. (QGIS has a character limit and sometimes doesn't like colons, so double check that `addr:house` is `addr:housenumber`, `addr:postc` is `addr:postcode`, `addr:stree` is `addr:street`, etc.)
* Mass-add new tags like `highway=residential`, `surface=asphalt`, etc, as indicated.
* Remove any spurious tags that may have been brought over in the import (if it's not in the OSM Wiki, we don't want it.)
* Press ctrl-shift-M to merge into the OSM data layer. There will be a warning, but click OK; we will be extra careful about validating the merge in the next steps.
* For addresses, remove any address nodes that seem to not reflect reality or be placed far from the street bearing their name: it's better to not have 123 Adams Street mapped at all, than to claim that 123 Adams Street is hovering over someone's newly-built house at 321 Franklin Avenue, 200 feet away from Adams Street. (Cities often won't remove old addresses, leading to confusion when new streets are built.)
* For roads, highlight multiple street segments which have the same name and press C to combine them: the county data has one way per road segment and that's excessive for OSM.
* Check the edges of the imported areas to ensure new roads are merged with any preexisting roads
* Check the import area to ensure no incorrect overlaps
* Use the JOSM validator to ensure no errors in imported data. Warnings about existing data separate from the import can be ignored.
* If there are duplicate house numbers in the data, investigate and remove the more-unlikely node or both nodes. For example `4650 Ramsell Road` is duplicated in the source data, but the easternmost copy is on the "odd" side of the street and between 4653 and 4663 so it's more likely to actually be 4651, 4655, 4657, 4659, or 4661. We have no way of knowing, so we can either delete it entirely or simply delete the housenumber tag and leave it as an address without a number for a future editor to review. (We may submit incomplete data, just not wrong data.) We then leave the westernmost copy alone since 4650 fits neatly in between 4640/4644 and 4654/4660.
* All known duplicates:
* 4886 C 472 (one is a daycare the other is a church)
* 5626 C THOMAS RD has unit numbers in the original data's Notes field
* 301 CURRY ST
* 401 HALL ST
* 340 HEALD WAY has many buildings and many units per building, in the notes
* 1908 LAUREL MANOR DR (one is a CELL TOWER)
* 1950 LAUREL MANOR DR (each one has multiple units in a range, in the notes)
* 6217 MEGGISON RD (one's note is "restroom/storage")
* 6221 MEGGISON RD (one's note is "pavilion")
* 6227 MEGGISON RD (one's note is "recreation center")
* 102 NE 4TH AVE (one is a cell tower, the other a water tower)
* 11750 NE 62ND TER (one's note is Pebble Springs retirement community building)
* 13813 NE 136TH LOOP UNIT 1306
* 8550 NE 138TH LN (each is a different building number)
* 4650 RAMSELL RD
* 400 RUTLAND ST (both say "church owned")
* 308 SHAWN AVE (one says Wildwood Acres, the other Progress Energy Pump 29)
* 413 S PINE ST
* 2605 TRILLIUM RDG (one says bldg 1, the other says meter)
* 2680 TRILLIUM RDG (bldg 3, meter, meter)
* 13940 US 441 (different building names in the notes)
* 702 WEBSTER ST (one is city of ww, the other retention pond)
* Click upload
* Make sure there are no erroneous Relations or other unwanted objects about to be uploaded.
* Use a descriptive changeset message like "Roads/Addresses in The Villages #villagesimport"
* Set the Source to be "Sumter County GIS"
* You can easily copy-paste the below into the Settings tab:
```
comment=Roads/Addresses in The Villages #villagesimport
import=yes
website=https://wiki.openstreetmap.org/wiki/The_Villages_Road_and_Address_Import
source=Sumter County GIS
source:url=https://gitlab.com/zyphlar/the-villages-import
``` ```
Note: The `--build` flag ensures the image is rebuilt with the latest code changes.
* Review imported data in Achavi or Osmcha to ensure it looks proper. 2. Access the web interface:
- Main interface: http://localhost:5000
- Map viewer: http://localhost:5000/map
3. Stop the container:
## Useful queries: ```bash
docker-compose down
```
[timeout:60];
area["name"="Florida"]->.state;
area["name"="Lake County"](area.state)->.searchArea;nwr["addr:housenumber"](area.searchArea);
(._;>;);
out meta;
``` ```
### Using Docker Directly
1. Build the image:
```bash
docker build -t osm-import-tools .
``` ```
[timeout:60];
area["name"="Florida"]->.state; 2. Run the container:
area["name"="Lake County"](area.state)->.searchArea;way["highway"](area.searchArea); ```bash
(._;>;); docker run -d \
out meta; -p 5000:5000 \
-v "$(pwd)/data:/data" \
--name osm-import-tools \
osm-import-tools
```
3. View logs:
```bash
docker logs -f osm-import-tools
```
4. Stop the container:
```bash
docker stop osm-import-tools
docker rm osm-import-tools
```
### Uploading to a registry
```bash
docker build -t osm-import-tools .
docker tag osm-import-tools your-registry.example.com/you/osm-import-tools
docker push your-registry.example.com/you/osm-import-tools
```
## Features
### Main Dashboard (/)
- Run data processing scripts for Lake and Sumter counties
- View real-time script output
- Access to:
- Diff Roads
- Diff Addresses
- Diff Multi-Use Paths
- Download OSM Data
### Map Viewer (/map)
- Interactive map viewer for GeoJSON files
- Upload and compare OSM, Diff, and County data
- Filter by removed/added features
- Hide highway=service roads
- Drag-and-drop layer reordering
- Click on features to view properties
- Accept/reject diff features
## Volume Mounts
The Docker container mounts a single data directory:
- `./data` → `/data` - All data files, organized by date
Inside `/data`, the structure is:
- `/data/latest/` - Symlink to the most recent data directory
- `/data/YYMMDD/lake/` - Lake County data for that date
- `/data/YYMMDD/sumter/` - Sumter County data for that date
All changes are persisted on the host in the local `./data` folder.
## API Endpoints
- `GET /` - Main dashboard
- `GET /map` - Map viewer
- `POST /api/run-script` - Execute a processing script
- `GET /api/job-status/<job_id>` - Get script status and logs
- `GET /api/list-files` - List available GeoJSON files
- `GET /data/<path>` - Serve GeoJSON files
## Troubleshooting
### Port already in use
If port 5000 is already in use, edit `docker-compose.yml`:
```yaml
ports:
- "8080:5000" # Change 8080 to any available port
```
### Permission issues
Ensure the data directory has proper permissions:
```bash
mkdir -p data
chmod -R 755 data
```
### View container logs
```bash
docker-compose logs -f
```
### Rebuild after code changes
```bash
docker-compose down
docker-compose build --no-cache
docker-compose up -d
``` ```
compare-addresses.py
+53 -12
View File
@@ -14,6 +14,7 @@ Usage:
import argparse import argparse
import json import json
import os import os
import re
import shutil import shutil
import sys import sys
import zipfile import zipfile
@@ -374,6 +375,37 @@ out geom;"""
return processed_gdf return processed_gdf
def _normalize_street_name(self, street: str) -> str:
"""
Normalize street names for better matching.
Handles common abbreviation and formatting differences between county and OSM data.
"""
if not street or street == 'nan':
return ''
street = street.strip()
# Convert to lowercase for comparison
street_lower = street.lower()
# Normalize State Road variations
street_lower = re.sub(r'\bstate road\b', 'sr', street_lower)
street_lower = re.sub(r'\bstate route\b', 'sr', street_lower)
# Normalize County Road variations
street_lower = re.sub(r'\bcounty road\b', 'cr', street_lower)
street_lower = re.sub(r'\bc\b', 'cr', street_lower) # "C 44a" -> "cr 44a"
# Normalize County Road number formatting: "109d 1" -> "109d-1", "109d-1" stays same
# This handles both space and hyphen separators
street_lower = re.sub(r'\b(cr\s+\d+[a-z])\s+(\d+)', r'\1-\2', street_lower)
street_lower = re.sub(r'\b(cr\s+\d+[a-z])-(\d+)', r'\1-\2', street_lower)
# Remove extra spaces
street_lower = re.sub(r'\s+', ' ', street_lower).strip()
return street_lower
def compare_addresses(self, local_file: str, osm_file: str) -> Tuple[List[Dict], List[Dict], List[Dict]]: def compare_addresses(self, local_file: str, osm_file: str) -> Tuple[List[Dict], List[Dict], List[Dict]]:
""" """
Compare local and OSM address data. Compare local and OSM address data.
@@ -435,19 +467,28 @@ out geom;"""
local_house_num = str(local_row.get('addr:housenumber', '')) local_house_num = str(local_row.get('addr:housenumber', ''))
osm_house_num = str(osm_row.get('addr:housenumber', '')) osm_house_num = str(osm_row.get('addr:housenumber', ''))
# Check street name match (required) # Check street name match (required) - use normalization for better matching
local_street = str(local_row.get('addr:street', '')).strip().lower() local_street = self._normalize_street_name(str(local_row.get('addr:street', '')))
osm_street = str(osm_row.get('addr:street', '')).strip().lower() osm_street = self._normalize_street_name(str(osm_row.get('addr:street', '')))
street_match = (local_street == osm_street and local_street != '' and local_street != 'nan') street_match = (local_street == osm_street and local_street != '')
# Check unit match (only if both have units specified) # Check unit match - if either has a unit, both must match
local_unit = local_row.get('addr:unit') local_unit = local_row.get('addr:unit')
osm_unit = osm_row.get('addr:unit') osm_unit = osm_row.get('addr:unit')
# Unit matches if: (1) both are None/null, OR (2) both have values and they're equal
unit_match = True # Determine if each side has a unit
if local_unit is not None and pd.notna(local_unit) and osm_unit is not None and pd.notna(osm_unit): local_has_unit = local_unit is not None and pd.notna(local_unit) and str(local_unit).strip() != ''
osm_has_unit = osm_unit is not None and pd.notna(osm_unit) and str(osm_unit).strip() != ''
if local_has_unit and osm_has_unit:
# Both have units - they must match # Both have units - they must match
unit_match = (str(local_unit).strip().lower() == str(osm_unit).strip().lower()) unit_match = (str(local_unit).strip().lower() == str(osm_unit).strip().lower())
elif local_has_unit or osm_has_unit:
# One has unit, other doesn't - no match
unit_match = False
else:
# Neither has unit - match
unit_match = True
# Only consider as potential match if house numbers, street, and unit all match # Only consider as potential match if house numbers, street, and unit all match
if (local_house_num == osm_house_num and if (local_house_num == osm_house_num and
@@ -500,10 +541,10 @@ out geom;"""
local_house_num = str(local_row.get('addr:housenumber', '')) local_house_num = str(local_row.get('addr:housenumber', ''))
osm_house_num = str(osm_row.get('addr:housenumber', '')) osm_house_num = str(osm_row.get('addr:housenumber', ''))
# Check street name match (required) # Check street name match (required) - use normalization for better matching
local_street = str(local_row.get('addr:street', '')).strip().lower() local_street = self._normalize_street_name(str(local_row.get('addr:street', '')))
osm_street = str(osm_row.get('addr:street', '')).strip().lower() osm_street = self._normalize_street_name(str(osm_row.get('addr:street', '')))
street_match = (local_street == osm_street and local_street != '' and local_street != 'nan') street_match = (local_street == osm_street and local_street != '')
# Check unit match (only if both have units specified) # Check unit match (only if both have units specified)
local_unit = local_row.get('addr:unit') local_unit = local_row.get('addr:unit')