Starintel 4th generation Ideas

Design File requirments

#+title: STAR-00 <short title>
#+description: <short description of the file>
,| Version  | Date  | Description of change  | Did nsaspy aprove it  |
,|---+---+---+---|
,|   |   |   |   |

* Design File System
* Foot Notes
* Citations

Agentic Design File scripts

a. FEC Data Actors

• TODO Supported collection methods
  
  • Official FEC API
    
  • Bulk data downloads
    
  • Public filing documents
    
  • Public amendment history
    
  • Public committee and candidate records
    
• TODO Required features
  
  • Incremental synchronization
    
  • Filing-version tracking
    
  • Donor and committee identity resolution
    
  • Address normalization
    
  • Amendment reconciliation
    
  • Citation and provenance generation
    

b. Melissa Data Actor

• TODO Research additional people-data providers
  
  • LexisNexis
    
  • Thomson Reuters CLEAR
    
  • TransUnion TLOxp
    
  • Tracers
    
  • Accurint
    
  • Experian
    
  • Equifax
    
  • Whitepages Pro
    
  • People Data Labs
    
  • FullContact
    
  • Verisk
    
  • Socure
    
  • IDI Data
    
  • Other data brokers
    

    Preferable if an normal joe individual can buy records (rank by price and data types)

• TODO Required features
  
  • Provider capability discovery
    
  • Per-provider credential records
    
  • Field-level provenance
    
  • Confidence scores
    
  • Cost and quota tracking
    
  • Result deduplication
    
  • Conflict detection
    
  • Restricted-field handling
    
  • Provider-specific retention rules
    

c. Telephone Number Actors

• TODO Capabilities
  
  • E.164 normalization
    
  • Country and region detection
    
  • Carrier lookup
    
  • Line-type detection
    
  • Porting-history lookup
    
  • Caller-name lookup
    
  • Reverse lookup
    
  • Reputation lookup
    
  • Breach and exposure correlation
    
  • User-supplied contact ingestion
    
  • File ingestion
    
• TODO Research providers
  
  • Twilio Lookup
    
  • Telnyx
    
  • Telesign
    
  • Vonage
    
  • Numverify
    
  • Abstract API
    
  • IPQualityScore
    
  • Whitepages Pro
    
  • Melissa
    
  • Carrier and CNAM providers
    

d. Bug-Bounty Tool Actors

• TODO Tool actors
  
  • Asset discovery
    
  • DNS enumeration
    
  • HTTP probing
    
  • Web crawling
    
  • Screenshot collection
    
  • Technology fingerprinting
    
  • Passive vulnerability matching
    
  • Active scanning
    
  • Result normalization
    
  • Scope validation
    
  • Duplicate-finding detection
    
  • Output packaging
    
• TODO Scope controls
  
  • Program scope ingestion
    
  • Domain and CIDR allowlists
    
  • Out-of-scope rejection
    
  • Per-tool safety policy
    
  • Request-rate limits
    
  • Target locks
    
  • Program-specific headers
    
  • Audit trail for every operation
    

e. Discord Self-Bot Collection Actors

• TODO Collection capabilities
  
  • Account and session handling
    
  • Guild traversal
    
  • Channel traversal
    
  • Thread traversal
    
  • Message collection
    
  • Reply and reaction collection
    
  • Member and role collection
    
  • Attachment and embed collection
    
  • Audit-log ingestion
    
  • Deleted-content ingestion when supplied by Discord
    
  • Incremental checkpoints
    
  • Historical backfill
    
  • Content preservation
    
• TODO Coordination requirements
  
  • Record Discord access reference
    
  • Record approved accounts
    
  • Record approved guilds and channels
    
  • Enforce collection boundaries
    
  • Respect Discord-supplied rate limits
    
  • Support access expiration
    
  • Preserve Discord-supplied data packages
    
  • Produce complete audit logs
    
  • Disable collection automatically when access expires
    

f. Telegram Collection Actors

• TODO Collection capabilities
  
  • Public channel collection
    
  • Public group collection
    
  • Private-group collection
    
  • Account history collection
    
  • Message and reply collection
    
  • Topic and thread collection
    
  • Member and administrator metadata
    
  • Forwarded-message relationships
    
  • Reactions and polls
    
  • Media and attachment collection
    
  • Edit and deletion tracking
    
  • Invite-link metadata
    
  • Incremental synchronization
    
• TODO Client support
  
  • Telegram Bot API
    
  • Telegram client API
    
  • User clients
    
  • User exports
    
  • Data packages
    
  • Source artifacts
    
• TODO Access controls
  
  • Session ownership records
    
  • Approved chat identifiers
    
  • Public/private source classification
    
  • Per-chat collection boundaries
    
  • Credential isolation
    
  • Session revocation
    
  • Rate-limit enforcement
    

g. Signal Messenger Collection Actors

• TODO Signal Mobile Actor
  

  Ingest Signal artifacts extracted from Android and iOS devices.

• TODO Signal Desktop Actor
  

  Parse Signal Desktop databases, configuration, attachments, account metadata, and locally retained history.

• TODO Signal Backup Actor
  

  Detect, validate, decrypt when keys or passphrases are available, and ingest supported Signal backup formats.

• TODO Signal Image Actor
  

  Locate Signal-related files and databases inside mounted images without modifying the source.

• TODO Signal Attachment Actor
  

  Extract images, video, audio, documents, stickers, contact cards, and other attachments while preserving message relationships.

• TODO Signal Conversation Actor
  

  Normalize recovered conversations, groups, calls, reactions, replies, edits, deletions, and disappearing-message remnants.

• TODO Signal Identity Actor
  

  Correlate account identifiers, usernames, phone numbers, profile keys, device records, contacts, and existing StarIntel identities.

• TODO Signal Group Actor
  

  Model group membership, administrators, membership changes, titles, avatars, invitations, and related data.

• TODO Signal Timeline Actor
  

  Construct an ordered timeline across messages, calls, attachments, device events, and account changes.

h. News-Ingestion Actors

• TODO Sources
  
  • RSS and Atom feeds
    
  • Publisher APIs
    
  • Licensed news APIs
    
  • Public news sites
    
  • Press-release services
    
  • Government newsrooms
    
  • NGO newsrooms
    
  • Corporate newsrooms
    
  • Broadcast transcripts
    
  • User-supplied archives
    
• TODO LLM-assisted document creation
  
  • Article classification
    
  • Entity extraction
    
  • Claim extraction
    
  • Event extraction
    
  • Date normalization
    
  • Location normalization
    
  • Source comparison
    
  • Duplicate detection
    
  • Contradiction detection
    
  • Summary generation
    
  • Citation generation
    
• TODO Required controls
  
  • Preserve raw source
    
  • Record publication and collection timestamps
    
  • Record licensing and retention metadata
    
  • Distinguish quoted claims from verified facts
    
  • Require structured LLM output
    
  • Require deterministic validation
    
  • Track model and prompt provenance
    

i. Scrapy Spider Actors

• TODO LLM-driven spider workflow
  
  • Receive a Target document
    
  • Inspect approved domains
    
  • Analyze page structure
    
  • Propose extraction rules
    
  • Generate a spider
    
  • Run static validation
    
  • Run against approved fixtures
    
  • Request human or policy approval
    
  • Execute with bounded scope
    
  • Emit typed documents
    
  • Preserve source snapshots
    
• TODO Required controls
  
  • Domain allowlists
    
  • Path allowlists
    
  • Depth limits
    
  • Request budgets
    
  • Rate limits
    
  • Robots and policy metadata
    
  • Authentication isolation
    
  • Duplicate-request detection
    
  • Content hashing
    
  • Spider versioning
    
  • Replayable fixtures
    
  • Automatic shutdown on scope violation
    

j. Playwright Actor

• TODO Capabilities
  
  • Browser-context isolation
    
  • Login flows
    
  • Page rendering
    
  • DOM extraction
    
  • Network-event capture
    
  • Download capture
    
  • Screenshot capture
    
  • PDF generation
    
  • Form interaction
    
  • Infinite-scroll handling
    
  • Client-side route traversal
    
  • Session replay
    
• TODO Required controls
  
  • Approved-domain enforcement
    
  • Approved-action policies
    
  • Credential isolation
    
  • Download quarantine
    
  • Popup and navigation controls
    
  • Request interception
    
  • Execution timeouts
    
  • Browser resource limits
    
  • Complete interaction logs
    
  • Deterministic replay where possible
    

k. Proxy-Rotation Actor

• TODO Capabilities
  
  • Proxy registration
    
  • Metadata
    
  • Health checks
    
  • Latency measurement
    
  • Geographic classification
    
  • Actor-specific allocation
    
  • Sticky-session support
    
  • Rotation policies
    
  • Failure detection
    
  • Quarantine
    
  • Expiration
    
  • Cost tracking
    
• TODO Restrictions
  
  • Bind every proxy to an approved project
    
  • Record all actor-to-proxy assignments
    
  • Enforce provider-specific limits
    
  • Disable expired or revoked proxies
    

l. Expert-System Actor Framework

• TODO Supported systems
  
  • Prolog
    
  • LISA
    
  • CLIPS-compatible systems
    
  • Datalog
    
  • Rete-based engines
    
  • Custom Common Lisp rule systems
    
• TODO Capabilities
  
  • Consume typed documents
    
  • Assert and retract facts
    
  • Run bounded inference
    
  • Emit conclusions
    
  • Emit proof traces
    
  • Emit confidence values
    
  • Request additional data
    
  • Participate in dataflows
    
  • Maintain isolated knowledge bases
    
  • Version rule sets
    
• TODO Required controls
  
  • Distinguish facts from conclusions
    
  • Preserve rule provenance
    
  • Record complete proof paths
    
  • Limit recursion and inference time
    
  • Prevent cross-case data leakage
    
  • Support deterministic replay
    
  • Require schema validation
    

m. Mastodon Actors

• TODO Capabilities
  
  • Instance discovery
    
  • Public timeline collection
    
  • Account collection
    
  • Status and reply collection
    
  • Boost and favorite relationships
    
  • Hashtag collection
    
  • Media collection
    
  • Profile collection
    
  • Follower and following collection where available
    
  • Moderation-event ingestion when supplied by an instance
    
  • Instance-block and federation metadata
    
  • Incremental synchronization
    
• TODO Access controls
  
  • Record instance rules
    
  • Record API credentials and scopes
    
  • Enforce per-instance rate limits
    
  • Track access expiration
    
  • Support instance-specific retention rules
    

n. Bluesky Actors

• TODO Capabilities
  
  • Repository ingestion
    
  • Firehose ingestion
    
  • Profile collection
    
  • Post and reply collection
    
  • Repost and like relationships
    
  • Follow relationships
    
  • Feed-generator data
    
  • Label and moderation metadata
    
  • Blob and media collection
    
  • DID resolution
    
  • Handle history
    
  • PLC directory history
    
  • Incremental synchronization
    
• TODO Required controls
  
  • Service and repository scope enforcement
    
  • DID-based identity preservation
    
  • Record source PDS
    
  • Record relay source
    
  • Validate signed repository data
    
  • Preserve commit and revision history
    
  • Track deletions and tombstones
    

o. 4chan and 8kun.top Actors

• TODO Capabilities
  
  • Board discovery
    
  • Catalog collection
    
  • Thread collection
    
  • Post collection
    
  • Reply relationships
    
  • Attachment collection
    
  • Thumbnail collection
    
  • Archive ingestion
    
  • Thread lifecycle tracking
    
  • Deletion detection
    
  • Tripcode and identifier extraction
    
  • Cross-thread entity correlation
    
• TODO Required controls
  
  • Board allowlists
    
  • Collection-rate limits
    
  • Source timestamps
    
  • Raw JSON or HTML preservation
    
  • Media hashing
    
  • Duplicate-media detection
    
  • Expired-thread handling
    
  • Archive provenance
    
  • Explicit content-handling policies
    

p. Reddit Collection Subsystem

• TODO Collection methods
  
  • Official Reddit API
    
  • Public feeds
    
  • old.reddit.com
    
  • Account exports
    
  • Moderator exports
    
  • Reddit-supplied data
    
  • Licensed archives
    
• TODO Capabilities
  
  • Subreddit collection
    
  • Post collection
    
  • Comment-tree collection
    
  • User-profile collection
    
  • Moderation-log ingestion
    
  • Flair collection
    
  • Award and vote metadata when available
    
  • Media and attachment collection
    
  • Edit and deletion tracking
    
  • Crosspost relationships
    
  • Incremental synchronization
    
• TODO Access controls
  
  • Record API application and scopes
    
  • Enforce subreddit allowlists
    
  • Support moderator access
    
  • Respect Reddit-supplied rate limits
    
  • Disable collection on access revocation
    
  • Preserve source and acquisition metadata
    

q. LinkedIn Actors

• TODO Supported sources
  
  • LinkedIn APIs
    
  • Organization-controlled company pages
    
  • Account-owner data exports
    
  • Recruiter or Sales Navigator exports where licensing permits
    
  • Job-posting feeds
    
  • Applicant and recruiting-system integrations
    
  • LinkedIn-supplied data packages
    
  • User-supplied records
    
• TODO Capabilities
  
  • Profile ingestion
    
  • Company-page ingestion
    
  • Employment-history normalization
    
  • Education-history normalization
    
  • Job-posting ingestion
    
  • Organization and person resolution
    
  • Connection and relationship ingestion when available
    
  • Post and article ingestion when available
    
  • Attachment and media ingestion
    
  • Incremental synchronization
    
• TODO Access controls
  
  • Record LinkedIn license
    
  • Record account-owner consent
    
  • Record approved organizations
    
  • Enforce API scopes
    
  • Enforce retention limits
    
  • Support access expiration
    
  • Preserve source exports unchanged
    
  • Reject out-of-scope profile or relationship collection
    

r. Shared Collection-Actor Requirements

• TODO Common actor features
  
  • Typed input and output documents
    
  • Actor manifests
    
  • Dataset manifests
    
  • Capability discovery
    
  • Dataflow integration
    
  • Pagination
    
  • Incremental checkpoints
    
  • Rate limiting
    
  • Backoff
    
  • Bounded concurrency
    
  • Target locks
    
  • Provenance
    
  • Source timestamps
    
  • Raw-response preservation
    
  • Deduplication
    
  • Attachment handling
    
  • Authentication isolation
    
  • Credential revocation
    
  • Health checks
    
  • Mock transports
    
  • Replay transports
    
  • Persistent documents
    
  • Transient documents
    
  • Structured errors
    
  • Dead-letter handling
    
  • Audit logs
    

expand all fields to cover as much metadata as possible

• TODO Stable document identifiers
  
• TODO Dataset identifiers
  
• TODO Document type identifiers
  
• TODO Schema identifiers
  
• TODO Schema version
  
• TODO Document version
  
• TODO Revision number
  
• TODO Creation timestamp
  
• TODO Ingestion timestamp
  
• TODO Observation timestamp
  
• TODO Publication timestamp
  
• TODO Last-seen timestamp
  
• TODO Last-updated timestamp
  
• TODO Expiration timestamp
  
• TODO Collector identity
  
• TODO Actor identity
  
• TODO Source identity
  
• TODO Source URL
  
• TODO Source mirror URLs
  
• TODO Archive URLs
  
• TODO Source access method
  
• TODO Source account
  
• TODO Source platform
  
• TODO Source dataset
  
• TODO Author
  
• TODO Publisher
  
• TODO Organization
  
• TODO Language
  
• TODO Locale
  
• TODO Country
  
• TODO Region
  
• TODO Timezone
  
• TODO Geographic coordinates
  
• TODO Confidence
  
• TODO Reliability
  
• TODO Relevance
  
• TODO Priority
  
• TODO Sensitivity
  
• TODO Classification
  
• TODO Access-control labels
  
• TODO Retention policy
  
• TODO Expiration policy
  
• TODO Deletion policy
  
• TODO Legal hold
  
• TODO Chain of custody
  
• TODO Provenance
  
• TODO Transformation history
  
• TODO Parent documents
  
• TODO Child documents
  
• TODO Related documents
  
• TODO Superseded documents
  
• TODO Duplicate documents
  
• TODO Near-duplicate documents
  
• TODO Content hashes
  
• TODO Fingerprints
  
• TODO Tags
  
• TODO Topics
  
• TODO Entities
  
• TODO Events
  
• TODO Claims
  
• TODO Relationships
  
• TODO Attachments
  
• TODO Citations
  
• TODO Processing state
  
• TODO Validation state
  
• TODO Review state
  
• TODO Error records
  
• TODO Warning records
  
• TODO Human edits
  
• TODO Human approvals
  
• TODO Actor configuration
  
• TODO Model metadata
  
• TODO Prompt metadata
  
• TODO Tool metadata
  
• TODO Pipeline metadata
  
• TODO Extension namespaces
  
• TODO Dataset-specific fields
  

a. unstructured docs

• TODO Plain text
  
• TODO Org
  
• TODO Markdown
  
• TODO HTML
  
• TODO XML
  
• TODO JSON
  
• TODO JSONL
  
• TODO CSV
  
• TODO Email
  
• TODO Chat logs
  
• TODO PDFs
  
• TODO Office documents
  
• TODO Reports
  
• TODO Transcripts
  
• TODO Scraped pages
  
• TODO OCR documents
  
• TODO Scanned documents
  
• TODO Preserve the original document
  
• TODO Preserve normalized text
  
• TODO Preserve extracted text
  
• TODO Preserve document structure
  
• TODO Headings
  
• TODO Sections
  
• TODO Paragraphs
  
• TODO Lists
  
• TODO Tables
  
• TODO Citations
  
• TODO Page numbers
  
• TODO Character offsets
  
• TODO Byte offsets
  
• TODO Bounding boxes
  
• TODO Source spans
  
• TODO Extracted entities
  
• TODO Extracted claims
  
• TODO Extracted events
  
• TODO Extracted quotations
  
• TODO Extracted relationships
  
• TODO Document summaries
  
• TODO Section summaries
  
• TODO Chunk summaries
  
• TODO Multiple translations
  
• TODO Redacted variants
  
• TODO Public-safe variants
  
• TODO Parsing errors
  
• TODO Partially processed documents
  
• TODO Chunking without losing hierarchy
  
• TODO Every derived chunk links to its original source span
  
• TODO Store embeddings separately from canonical document data
  

b. custom view code (with security mitigation)

• TODO Dataset-specific renderers
  
• TODO Document-specific renderers
  
• TODO Table views
  
• TODO Graph views
  
• TODO Timeline views
  
• TODO Map views
  
• TODO Gallery views
  
• TODO Evidence views
  
• TODO Declarative views
  
• TODO Server-side views
  
• TODO Client-side views
  
• TODO Common Lisp views
  
• TODO JavaScript views
  
• TODO WebAssembly views
  
• TODO Sandboxed execution
  
• TODO Capability-based permissions
  
• TODO Read-only data access by default
  
• TODO No network access by default
  
• TODO No filesystem access by default
  
• TODO No process execution by default
  
• TODO No secret access by default
  
• TODO CPU limits
  
• TODO Memory limits
  
• TODO Execution-time limits
  
• TODO Output-size limits
  
• TODO Dependency allowlists
  
• TODO Dependency lockfiles
  
• TODO Signed view packages
  
• TODO Versioned view packages
  
• TODO Static analysis before installation
  
• TODO Content Security Policy
  
• TODO HTML sanitization
  
• TODO Output escaping
  
• TODO Per-view field allowlists
  
• TODO Per-view dataset allowlists
  
• TODO Audit every execution
  
• TODO Administrative approval
  
• TODO Emergency disable controls
  
• TODO Safe fallback renderer
  
• TODO Deterministic rendering where possible
  

c. attachments and files

• TODO Attach files to documents
  
• TODO Attach files to entities
  
• TODO Attach files to events
  
• TODO Attach files to cases
  
• TODO Original filename
  
• TODO Normalized filename
  
• TODO MIME type
  
• TODO Detected file type
  
• TODO File size
  
• TODO Cryptographic hashes
  
• TODO Fuzzy hashes
  
• TODO Content-addressed storage
  
• TODO Deduplication
  
• TODO Versioning
  
• TODO Revision history
  
• TODO Archive support
  
• TODO Compressed-file support
  
• TODO Image previews
  
• TODO Audio previews
  
• TODO Video previews
  
• TODO Document previews
  
• TODO Source-code previews
  
• TODO Thumbnail generation
  
• TODO Derivative generation
  
• TODO Text extraction
  
• TODO Metadata extraction
  
• TODO Embedding extraction
  
• TODO Malware scanning
  
• TODO Quarantine
  
• TODO Encryption at rest
  
• TODO Per-file access control
  
• TODO Retention policy
  
• TODO Deletion policy
  
• TODO Legal holds
  
• TODO Evidence-preservation flags
  
• TODO Chain-of-custody records
  
• TODO External object-storage references
  
• TODO Multipart uploads
  
• TODO Resumable uploads
  
• TODO Download limits
  
• TODO Bandwidth limits
  
• TODO Inline attachment references
  
• TODO Attachment relationships
  
• TODO Verify integrity during storage
  
• TODO Verify integrity during retrieval
  

d. support for expert systems

• TODO Facts
  
• TODO Rules
  
• TODO Predicates
  
• TODO Frames
  
• TODO Ontologies
  
• TODO Prolog
  
• TODO LISA
  
• TODO CLIPS-compatible systems
  
• TODO Datalog
  
• TODO Rete-based systems
  
• TODO Custom Common Lisp rule systems
  
• TODO Dataset-to-fact transformation rules
  
• TODO Namespaced predicates
  
• TODO Namespaced vocabularies
  
• TODO Forward chaining
  
• TODO Backward chaining
  
• TODO Explanation traces
  
• TODO Proof traces
  
• TODO Confidence-weighted facts
  
• TODO Contradiction detection
  
• TODO Conflict detection
  
• TODO Temporal facts
  
• TODO Interval reasoning
  
• TODO Geospatial facts
  
• TODO Relationship reasoning
  
• TODO Rule versioning
  
• TODO Rule migration
  
• TODO Rule activation
  
• TODO Rule deactivation
  
• TODO Incremental fact updates
  
• TODO Materialized conclusions
  
• TODO Separate observed facts
  
• TODO Separate asserted facts
  
• TODO Separate inferred facts
  
• TODO Human confirmation
  
• TODO Fact correction
  
• TODO Query budgets
  
• TODO Recursion limits
  
• TODO Inference-time limits
  
• TODO Sandboxed custom predicates
  
• TODO Rule dependency graphs
  
• TODO Dataset dependency graphs
  
• TODO Export conclusions back into documents
  
• TODO Store exact rules used for each conclusion
  
• TODO Store exact evidence used for each conclusion
  

e. SIMHASH/MINHASH/JARM/custom finger print

• TODO SimHash
  
• TODO MinHash
  
• TODO TLSH
  
• TODO ssdeep
  
• TODO JARM
  
• TODO JA3
  
• TODO JA3S
  
• TODO JA4
  
• TODO HTTP fingerprints
  
• TODO Header-order fingerprints
  
• TODO DOM fingerprints
  
• TODO Favicon hashes
  
• TODO Certificate fingerprints
  
• TODO SSH host-key fingerprints
  
• TODO Service-banner fingerprints
  
• TODO File-structure fingerprints
  
• TODO Image perceptual hashes
  
• TODO Audio fingerprints
  
• TODO Video fingerprints
  
• TODO Source-code fingerprints
  
• TODO Dataset-defined fingerprint algorithms
  
• TODO Fingerprint algorithm version
  
• TODO Fingerprint parameters
  
• TODO Fingerprint confidence
  
• TODO Fingerprint provenance
  
• TODO Multiple fingerprints per document
  
• TODO Fingerprint comparison records
  
• TODO Threshold configuration
  
• TODO Approximate nearest-neighbor indexes
  

f. Uiniqly Fingerprint a document

• TODO Canonical document representation
  
• TODO Canonical field ordering
  
• TODO Canonical whitespace normalization
  
• TODO Canonical Unicode normalization
  
• TODO Canonical timestamp normalization
  
• TODO Canonical URL normalization
  
• TODO Canonical attachment ordering
  
• TODO Canonical relationship ordering
  
• TODO Exclude volatile metadata
  
• TODO Include schema version
  
• TODO Include document type
  
• TODO Include dataset namespace
  
• TODO Strong cryptographic document hash
  
• TODO Stable semantic fingerprint
  
• TODO Structural fingerprint
  
• TODO Content fingerprint
  
• TODO Source fingerprint
  
• TODO Attachment-set fingerprint
  
• TODO Revision fingerprint
  
• TODO Cross-format equivalence detection
  
• TODO Exact-duplicate detection
  
• TODO Near-duplicate detection
  
• TODO Collision handling
  
• TODO Fingerprint migration
  
• TODO Fingerprint verification
  
• TODO Fingerprint audit history
  

h. Speficic Stable id for people if entity too brad

• TODO Person-specific stable identifier
  
• TODO Never use a display name as the stable identifier
  
• TODO Preserve all aliases
  
• TODO Preserve former names
  
• TODO Preserve usernames
  
• TODO Preserve telephone numbers
  
• TODO Preserve email addresses
  
• TODO Preserve account identifiers
  
• TODO Preserve external provider identifiers
  
• TODO Preserve government identifiers only in restricted fields
  
• TODO Preserve source-specific identifiers
  
• TODO Identifier namespace
  
• TODO Identifier type
  
• TODO Identifier issuer
  
• TODO Identifier validity interval
  
• TODO Identifier confidence
  
• TODO Identifier provenance
  
• TODO Identity-resolution evidence
  
• TODO Identity-resolution confidence
  
• TODO Competing identity hypotheses
  
• TODO Reversible entity merges
  
• TODO Entity split history
  
• TODO Entity merge history
  
• TODO Never destroy source records during a merge
  
• TODO Person-to-account relationships
  
• TODO Person-to-organization relationships
  
• TODO Person-to-location relationships
  
• TODO Person-to-device relationships
  
• TODO Temporal identity changes
  

j. Confidence, reliability, relevance, and priority scores

• TODO Separate every score
  
• TODO Score range definitions
  
• TODO Confidence in extracted value
  
• TODO Confidence in entity resolution
  
• TODO Confidence in inferred relationship
  
• TODO Confidence in expert-system conclusion
  
• TODO Source reliability
  
• TODO Collector reliability
  
• TODO Actor reliability
  
• TODO Model reliability
  
• TODO Document relevance
  
• TODO Field relevance
  
• TODO Case relevance
  
• TODO Search-result relevance
  
• TODO Processing priority
  
• TODO Review priority
  
• TODO Alert priority
  
• TODO Severity
  
• TODO Urgency
  
• TODO Score provenance
  
• TODO Scoring algorithm
  
• TODO Scoring algorithm version
  
• TODO Scoring inputs
  
• TODO Human overrides
  
• TODO Override reason
  
• TODO Score history
  
• TODO Calibration data
  
• TODO Uncertainty intervals
  
• TODO Missing-data indicators
  
• TODO Do not collapse unrelated scores into one number
  

k. Schema version and migration history

• TODO Version every schema
  
• TODO Version every document type
  
• TODO Version every dataset manifest
  
• TODO Version every actor manifest
  
• TODO Forward compatibility
  
• TODO Backward compatibility
  
• TODO Schema registry
  
• TODO Migration registry
  
• TODO Automated migrations
  
• TODO Manual migrations
  
• TODO Dry-run migrations
  
• TODO Per-dataset migration state
  
• TODO Per-document migration state
  
• TODO Migration checkpoints
  
• TODO Migration rollback
  
• TODO Migration validation
  
• TODO Migration error records
  
• TODO Migration provenance
  
• TODO Migration code version
  
• TODO Deprecated-field lifecycle
  
• TODO Field rename history
  
• TODO Field split history
  
• TODO Field merge history
  
• TODO Type-change history
  
• TODO Preserve unknown extension fields
  
• TODO Preserve original pre-migration document
  
• TODO Generate migration reports
  

i. Facial Reconigition Support

• TODO Face detection
  
• TODO Face embeddings
  
• TODO Face clustering
  
• TODO Face comparison
  
• TODO Face-search actors
  
• TODO Image ingestion
  
• TODO Video-frame ingestion
  
• TODO Keyframe extraction
  
• TODO Multiple faces per frame
  
• TODO Bounding boxes
  
• TODO Landmarks
  
• TODO Pose
  
• TODO Occlusion
  
• TODO Image quality
  
• TODO Detection confidence
  
• TODO Match confidence
  
• TODO Model identity
  
• TODO Model version
  
• TODO Embedding version
  
• TODO Comparison threshold
  
• TODO Candidate rankings
  
• TODO Human review state
  
• TODO Confirmed match state
  
• TODO Rejected match state
  
• TODO Unknown match state
  
• TODO Reference-image provenance
  
• TODO Source-image provenance
  
• TODO Frame timestamp
  
• TODO Track identity across video frames
  
• TODO Link faces to person hypotheses
  
• TODO Preserve competing person hypotheses
  
• TODO Facial-recognition actor support
  
• TODO Vector indexes partitioned by dataset
  
• TODO Support replacing models without destroying old embeddings
  

l. Expansive Support For Bug bounty, recon, and cyber security document types

• TODO Program
  
• TODO Scope
  
• TODO Target
  
• TODO Domain
  
• TODO Subdomain
  
• TODO IP address
  
• TODO CIDR
  
• TODO ASN
  
• TODO URL
  
• TODO Endpoint
  
• TODO Service
  
• TODO Port
  
• TODO Protocol
  
• TODO DNS record
  
• TODO Certificate
  
• TODO TLS configuration
  
• TODO HTTP response
  
• TODO HTTP request
  
• TODO WebSocket traffic
  
• TODO Proxy traffic
  
• TODO Screenshot
  
• TODO Technology
  
• TODO Product
  
• TODO Product version
  
• TODO CPE
  
• TODO CVE
  
• TODO CWE
  
• TODO CAPEC
  
• TODO Vulnerability
  
• TODO Finding
  
• TODO Evidence
  
• TODO Reproduction steps
  
• TODO Exploitability
  
• TODO Impact
  
• TODO Severity
  
• TODO CVSS
  
• TODO EPSS
  
• TODO KEV status
  
• TODO Remediation
  
• TODO Retest result
  
• TODO Duplicate finding
  
• TODO False positive
  
• TODO Accepted risk
  
• TODO Disclosure status
  
• TODO Bounty submission
  
• TODO Program response
  
• TODO Tool execution
  
• TODO Command provenance
  
• TODO Scan
  
• TODO Scan configuration
  
• TODO Scan result
  
• TODO Wordlist
  
• TODO Payload
  
• TODO Request template
  
• TODO Response template
  
• TODO Authentication material reference
  
• TODO Secret finding
  
• TODO Credential finding
  
• TODO Cloud asset
  
• TODO Container image
  
• TODO Package
  
• TODO Dependency
  
• TODO Repository
  
• TODO Commit
  
• TODO Branch
  
• TODO Pull request
  
• TODO Issue
  
• TODO Build
  
• TODO CI run
  
• TODO Artifact
  
• TODO SBOM
  
• TODO Malware sample
  
• TODO IOC
  
• TODO TTP
  
• TODO ATT&CK technique
  
• TODO Campaign
  
• TODO Threat actor
  
• TODO Incident
  
• TODO Timeline event
  
• TODO Scope validation state
  
• TODO Target lock
  
• TODO Rate-limit state
  
• TODO Raw tool output
  
• TODO Normalized tool output
  
• TODO Structured parser errors
  
• TODO Tool version
  
• TODO Tool configuration hash
  
• TODO Complete audit trail
  

m. Entity and relationship model

• TODO People
  
• TODO Organizations
  
• TODO Accounts
  
• TODO Locations
  
• TODO Devices
  
• TODO Documents
  
• TODO Events
  
• TODO Cases
  
• TODO Typed relationships
  
• TODO Directional relationships
  
• TODO Relationship confidence
  
• TODO Relationship provenance
  
• TODO Relationship evidence
  
• TODO Alias history
  
• TODO Identity-resolution history
  
• TODO Merge history
  
• TODO Split history
  
• TODO Competing relationship hypotheses
  
• TODO Temporal relationship validity
  
• TODO Dataset-specific entity extensions
  
• TODO Graph traversal limits
  

n. Event model

• TODO Point-in-time events
  
• TODO Interval events
  
• TODO Estimated timestamps
  
• TODO Uncertain timestamps
  
• TODO Recurring events
  
• TODO Participants
  
• TODO Locations
  
• TODO Causes
  
• TODO Outcomes
  
• TODO Evidence
  
• TODO Event correlation
  
• TODO Event deduplication
  
• TODO Timelines
  
• TODO Source disagreement
  
• TODO Derived events
  
• TODO Event severity
  
• TODO Event priority
  
• TODO Document links
  
• TODO Entity links
  
• TODO Attachment links
  

o. Provenance and lineage

• TODO Record every collector
  
• TODO Record every transformation
  
• TODO Preserve original representations
  
• TODO Preserve derived representations
  
• TODO Actor version
  
• TODO Tool version
  
• TODO Model version
  
• TODO Prompt version
  
• TODO Code version
  
• TODO Configuration version
  
• TODO Input hashes
  
• TODO Output hashes
  
• TODO Reproducible processing runs
  
• TODO Manual edit records
  
• TODO Source deletion records
  
• TODO Source disappearance records
  
• TODO Broken-source state
  
• TODO Unverifiable-source state
  
• TODO Provenance graph queries
  
• TODO Export complete evidence bundles
  

p. Search and retrieval

• TODO Full-text search
  
• TODO Structured field search
  
• TODO Graph search
  
• TODO Semantic vector search
  
• TODO Hybrid ranking
  
• TODO Temporal search
  
• TODO Geospatial search
  
• TODO Exact source-span retrieval
  
• TODO Saved queries
  
• TODO Query explanations
  
• TODO Per-dataset indexes
  
• TODO Index versioning
  
• TODO Index rebuild state
  
• TODO Search-result confidence
  
• TODO Search-result provenance
  
• TODO Permission-aware indexing
  

q. Validation and data quality

• TODO Required fields
  
• TODO Optional fields
  
• TODO Computed fields
  
• TODO Forbidden fields
  
• TODO Type validation
  
• TODO Range validation
  
• TODO Format validation
  
• TODO Relationship validation
  
• TODO Dataset-specific validation
  
• TODO Duplicate detection
  
• TODO Near-duplicate detection
  
• TODO Conflicting-value detection
  
• TODO Missing-source detection
  
• TODO Staleness indicators
  
• TODO Human-review queues
  
• TODO Quality scoring
  
• TODO Repair suggestions
  
• TODO Quarantine invalid documents without deleting them
  

r. Interchange and export

• TODO JSON
  
• TODO JSON-LD
  
• TODO RDF
  
• TODO CSV
  
• TODO Org
  
• TODO Markdown
  
• TODO Archive bundles
  
• TODO Dataset snapshots
  
• TODO Incremental exports
  
• TODO Signed exports
  
• TODO Portable evidence packages
  
• TODO Import validation
  
• TODO Import conflict handling
  
• TODO Preserve identifiers across systems
  
• TODO Export redacted variants
  
• TODO Streaming import
  
• TODO Streaming export
  
• TODO External schema mappings
  

s. Case and investigation containers

• TODO Group documents
  
• TODO Group entities
  
• TODO Group events
  
• TODO Group files
  
• TODO Group conclusions
  
• TODO Case-specific access controls
  
• TODO Notes
  
• TODO Tasks
  
• TODO Hypotheses
  
• TODO Review states
  
• TODO Evidence numbering
  
• TODO Chain of custody
  
• TODO Exportable case bundles
  
• TODO Case templates
  
• TODO Cross-case linking
  
• TODO Prevent automatic cross-case data leakage
  
• TODO Preserve investigator annotations separately from source data
  

t. Annotation system

• TODO Human annotations
  
• TODO Machine annotations
  
• TODO Span annotations
  
• TODO Field annotations
  
• TODO Document annotations
  
• TODO Entity annotations
  
• TODO Relationship annotations
  
• TODO Annotation author
  
• TODO Annotation timestamp
  
• TODO Annotation confidence
  
• TODO Annotation review state
  
• TODO Competing annotations
  
• TODO Correction history
  
• TODO Supersession history
  
• TODO Annotation schemas
  
• TODO Convert approved annotations into facts
  
• TODO Convert approved annotations into training data
  

u. Reproducibility and snapshots

• TODO Immutable dataset snapshots
  
• TODO Snapshot manifests
  
• TODO Snapshot hashes
  
• TODO Pin schema versions
  
• TODO Pin actor versions
  
• TODO Pin model versions
  
• TODO Pin rule versions
  
• TODO Pin dependency versions
  
• TODO Re-run historical queries against historical state
  
• TODO Compare snapshots
  
• TODO Record non-reproducible external dependencies
  
• TODO Generate reproducibility reports
  

a. Lisp actor library

b. Python Actor Library

c. Nim Based lmdb database based entirly on tek-9 called star-actor cache. which will be used in python, lisp, other langs that have c calling functionality. it will be a full database, but for most use cases will be a cache

a. Review design of gserver actor system and how cl-gserver intends actor systems to be made

k. Review latest cl-gserver code and improvments that can be applied after all bug fixes are made

b. Dead code drop

c. document validation

d. Target Locks

e. normalizing text and inputs

f. Actor Manifest support

g. Dataset Manifest support (the flow based programming backend for directing docuemnts to actors) MUST use the orignal ibm guy idea of flow based programming.

k. database agnostic protocols (actors across languages will use it, via json when sexp not avail)

l. data wearhousing scheama

m. prolog Based graph Search

n. Full p2p system with libp2p

o. Fully consider going back to zmq <starRouter> but in common lisp

p. Full StarIntel-Centered DSL

• TODO Common Lisp
  

  Attempt first. Use ordinary Lisp syntax, macros, CLOS, generic functions, and condition handling. Do not use reader macros.

• TODO Racket
  

  Implement as both:

  • TODO An embedded Racket DSL
    
  • TODO A dedicated #lang starintel
    
• TODO Scheme
  

  Implement the same DSL using portable Scheme syntax where possible. Test at least one practical implementation such as Chez Scheme, Guile, or Gambit.

• TODO Comparative Research Loop
  

  The research agent must implement the same example workflow in every language.

  Each prototype must:

  • TODO Use the same document schemas
    
  • TODO Use the same actor behaviors
    
  • TODO Use the same agent and tool definitions
    
  • TODO Use the same dataflow
    
  • TODO Use transient and persistent documents
    
  • TODO Produce inspectable macro expansions or compiled forms
    
  • TODO Include runnable tests
    
  • TODO Record syntax advantages and limitations
    
  • TODO Compare implementation complexity
    
  • TODO Compare debugging and editor support
    
  • TODO Compare runtime performance
    
  • TODO Compare interoperability with existing StarIntel components
    
  • TODO Recommend a language only after all prototypes are complete
    

    Do not select a language based only on appearance.

• TODO Required Example Workflow
  

  The comparison prototype must implement this complete workflow:

  • TODO Receive a Target document
    
  • TODO Match target.options.enumeration = true
    
  • TODO Require target.data to contain a User
    
  • TODO Extract the user's username
    
  • TODO Append several common email domains
    
  • TODO Emit one transient email-candidate document per domain
    
  • TODO Send each candidate to a testing actor
    
  • TODO Use a fake testing implementation during DSL research
    
  • TODO Submit found candidates to an LLM-native agent
    
  • TODO Allow the agent to call declared tools
    
  • TODO Persist only the final review document
    
  • TODO Express the entire process as a named dataflow
    
• Minimum DSL Surface
  

  The following names describe required language operations. The exact spelling may vary slightly between implementations, but every prototype must provide equivalent behavior.

  • Definition Forms
    
    • define-document
      

      Define a typed StarIntel document schema.

    • define-actor
      

      Define an actor, its accepted documents, matching rules, and behavior.

    • define-agent
      

      Define an LLM-native actor with a runtime-provided agent loop.

    • define-tool
      

      Define a typed tool available to an agent.

    • define-dataflow
      

      Define a named graph connecting actors, agents, sources, and sinks.

  • Document Forms
    
    • document
      

      Create and validate a document instance.

    • transient-document
      

      Create a document that may move through a dataflow but must not be persisted.

    • document-ref
      

      Read a field using a structured path.

      Example:

      (document-ref target 'data 'username)
      

    • document-type-p
      

      Test whether a value is a specific document type.

    • validate-document
      

      Validate a document against its registered schema.

  • Actor Forms
    
    • receive
      

      Declare or wait for accepted input.

    • send
      

      Send a document to a specific actor.

    • ask
      

      Send a request and receive a reply.

    • emit
      

      Publish a document into the current dataflow.

    • find-actor
      

      Find actors by name, capability, accepted document type, dataset, or availability.

    • actor-ref
      

      Return a stable actor reference.

  • Agent Forms
    
    • agent-loop
      

      Run the standard StarIntel LLM agent loop.

      The runtime must provide:

      • Tool selection
        
      • Tool invocation
        
      • Tool-result insertion
        
      • Step limits
        
      • Retry limits
        
      • Structured-output validation
        
      • Completion detection
        
      • Event logging
        
    • call-tool
      

      Invoke a declared agent tool.

    • complete
      

      Request a final structured result from the configured model.

    • agent-result
      

      Return the agent's final typed document.

  • Dataflow Forms
    
    • from
      

      Declare the input source or accepted document type.

    • through
      

      Route documents through an actor or agent stage.

    • branch
      

      Route documents conditionally.

    • filter
      

      Allow only matching documents to continue.

    • map
      

      Transform one input document into one output document.

    • flat-map
      

      Transform one input document into multiple output documents.

    • parallel
      

      Run a stage with bounded concurrency.

    • merge
      

      Combine multiple branches.

    • into
      

      Declare a sink.

    • run-dataflow
      

      Start a named dataflow with an input document.

  • Persistence Forms
    
    • persist
      

      Save a persistent document.

    • delete-document
      

      Delete a stored document.

    • couch-get
      

      Retrieve a CouchDB document by ID.

    • couch-put
      

      Create or update a CouchDB document.

    • couch-find
      

      Run a Mango query.

    • couch-view
      

      Query a CouchDB view.

    • couch-changes
      

      Consume the CouchDB changes feed.

    • couch-attach
      

      Add an attachment to a document.

  • Required Symbols and Values
    
    • persistent
      

      Document may be stored.

    • transient
      

      Document must not be stored.

    • actor
      

      A normal actor stage.

    • agent
      

      An LLM-native actor stage.

    • source
      

      A dataflow input.

    • sink
      

      A dataflow output.

    • capability
      

      A declared actor or tool capability.

    • document-type
      

      A registered StarIntel document type.

    • dataset
      

      A registered StarIntel dataset.

    • found
      

      Candidate was found.

    • not-found
      

      Candidate was not found.

    • unknown
      

      Candidate could not be conclusively tested.

• Minimum Type System
  

  The DSL must support:

  • Primitive types
    
    • String
      
    • Boolean
      
    • Integer
      
    • Number
      
    • Symbol
      
    • UUID
      
    • Timestamp
      
    • URI
      
    • Email address
      
    • Byte sequence
      
    • Arbitrary JSON-compatible value
      
  • Compound types
    
    • Optional values
      
    • Enumerations
      
    • Lists
      
    • Maps
      
    • Nested documents
      
    • References to other documents
      
    • Attachments
      
  • Field options
    
    • Required
      
    • Optional
      
    • Default value
      
    • Enumeration
      
    • Validator
      
    • Persistence policy
      
    • Indexed field
      
    • Secret or redacted field
      
• Minimum Runtime Requirements
  
  • TODO Actor registry
    
  • TODO Capability-based actor discovery
    
  • TODO Typed actor mailboxes
    
  • TODO In-process transport
    
  • TODO RabbitMQ transport
    
  • TODO Dataflow scheduler
    
  • TODO Bounded parallelism
    
  • TODO Backpressure
    
  • TODO Retry policies
    
  • TODO Timeouts
    
  • TODO Cancellation
    
  • TODO Dead-letter handling
    
  • TODO Document provenance
    
  • TODO Dataflow trace IDs
    
  • TODO Structured errors
    
  • TODO CouchDB integration
    
  • TODO LLM provider abstraction
    
  • TODO Tool-call validation
    
  • TODO Deterministic test runtime
    
• Required Research Artifacts
  

  For each implementation, create:

  • TODO Syntax examples
    
  • TODO Runnable prototype
    
  • TODO Macro-expansion examples
    
  • TODO Runtime architecture notes
    
  • TODO Test results
    
  • TODO Benchmark results
    
  • TODO Interoperability notes
    
  • TODO Advantages
    
  • TODO Disadvantages
    
  • TODO Unresolved problems
    

    The final comparison document must rank:

  • TODO Common Lisp
    
  • TODO Racket
    
  • TODO Scheme
    

    The ranking must separately score:

  • Syntax
    
  • Extensibility
    
  • Macro system
    
  • Type and schema support
    
  • Actor implementation
    
  • Agent implementation
    
  • Dataflow implementation
    
  • CouchDB integration
    
  • LLM integration
    
  • Debuggability
    
  • Performance
    
  • Deployment
    
  • Existing StarIntel compatibility** Languages
    

q. Possible port to Scheme + Scheme Actors

Design custom zmq protocol for couchdb

Research how to package FTS server

A wrapping http service with internal zmq eventing

Modify couchdb to accept ZMQ protocol

Implement a couchdb internal graph database

Support Cypher Query

Support