osm-import-tools/README.md

# OSM Import Tools

## Using

### Roads

### Addresses

Run after Roads to reduce validation errors. Easiest to hide existing (gray) and removed (red) and just focus on what new addresses to add.

Open in JOSM, and download the affected areas from OSM including highways with this Overpass query:

### Paths

Only Sumter currently has paths available.

## Docker Quick Start

### Using Docker Compose (Recommended)

1. Build and start the container:
```bash
docker-compose up -d --build
```

Note: The `--build` flag ensures the image is rebuilt with the latest code changes.

2. Access the web interface:
- Main interface: http://localhost:5000
- Map viewer: http://localhost:5000/map

3. Stop the container:
```bash
docker-compose down
```

### Using Docker Directly

1. Build the image:
```bash
docker build -t osm-import-tools .
```

2. Run the container:
```bash
docker run -d \
  -p 5000:5000 \
  -v "$(pwd)/data:/data" \
  --name osm-import-tools \
  osm-import-tools
```

3. View logs:
```bash
docker logs -f osm-import-tools
```

4. Stop the container:
```bash
docker stop osm-import-tools
docker rm osm-import-tools
```

### Deploying with Portainer

Just push to git and set up a pull from git; Portainer can build and run the project directly.

### Uploading to a registry
```bash
docker build -t osm-import-tools .
docker tag osm-import-tools your-registry.example.com/you/osm-import-tools
docker push your-registry.example.com/you/osm-import-tools
```

## Features

### Main Dashboard (/)
- Run data processing scripts for Lake and Sumter counties
- View real-time script output
- Access to:
  - Diff Roads
  - Diff Addresses
  - Diff Multi-Use Paths
  - Download OSM Data

### Map Viewer (/map)
- Interactive map viewer for GeoJSON files
- Upload and compare OSM, Diff, and County data
- Filter by removed/added features
- Hide highway=service roads
- Drag-and-drop layer reordering
- Click on features to view properties
- Accept/reject diff features

## Volume Mounts

The Docker container mounts a single data directory:
- `./data` → `/data` - All data files, organized by date

Inside `/data`, the structure is:
- `/data/latest/` - Symlink to the most recent data directory
- `/data/YYMMDD/lake/` - Lake County data for that date
- `/data/YYMMDD/sumter/` - Sumter County data for that date

All changes are persisted on the host in the local `./data` folder.

## API Endpoints

- `GET /` - Main dashboard
- `GET /map` - Map viewer
- `POST /api/run-script` - Execute a processing script
- `GET /api/job-status/<job_id>` - Get script status and logs
- `GET /api/list-files` - List available GeoJSON files
- `GET /data/<path>` - Serve GeoJSON files

## Troubleshooting

### Port already in use
If port 5000 is already in use, edit `docker-compose.yml`:
```yaml
ports:
  - "8080:5000"  # Change 8080 to any available port
```

### Permission issues
Ensure the data directory has proper permissions:
```bash
mkdir -p data
chmod -R 755 data
```

### View container logs
```bash
docker-compose logs -f
```

### Rebuild after code changes
```bash
docker-compose down
docker-compose build --no-cache
docker-compose up -d
```

## TODO

- Validate whether a similar address is within 50 feet (same number?)
- If addr:unit is like "Apartment 123" or "Apartment 123;Apartment 124;Apartment 125" remove the word Apartment so it's just "123" or "123;124;125"
- If an item gets through processing and is "<Null>" it should be cleared
- If a value is a range like "123 - 456" assume it's the same as "123-456" for matching purposes
- Unit numbers with semicolons are still getting detected as different despite being identical, like:
addr:city=Wildwood
addr:housenumber=8655
addr:postcode=34785
addr:state=FL
addr:street=Easterly Lane
addr:unit=108;208;308
- Some items appear duplicate in the data itself for unknown reasons:
addr:city=Bushnell
addr:housenumber=5740
addr:postcode=33513
addr:state=FL
addr:street=Hideaway Circle
- Sr is SR (State Route)



### Corrections

{
  "corrections": [
    {
      "from": "D Angelo Lane",
      "to": "D'Angelo Lane"
    },
    {
      "from": "Lajolla Circle",
      "to": "La Jolla Circle"
    },
    {
      "from": "Gardena Court",
      "to": "Gardenia Court"
    },
    {
      "from": "Glenmont Court",
      "to": "Glenmount Court"
    },
    {
      "from": "Pawleys Island Path",
      "to": "Pawley's Island Path"
    },
    {
      "from": "Oday Street",
      "to": "O'Day Street"
    },
    {
      "from": "Obrien Place",
      "to": "O'Brien Place"
    },
    {
      "from": "Ohara Court",
      "to": "O'Hara Court"
    },
    {
      "from": "Drive Randy Mcdaniel Way",
      "to": "Doctor Randy McDaniel Way"
    },
    {
      "from": "Mcneillrescue Lane",
      "to": "Rescue Lane"
    },
    {
      "from": "Longleaf Lane",
      "to": "Long Leaf Lane"
    }
  ]
}