Database design 基于elasticsearch的消息传递系统

我们正在构建一个消息传递系统，其中消息分布在多个列表之间。我们当前的实现是基于redis的，看起来像这样：

Message comes in from user “peeter”
Message is added to list “user:peeter:messages”
Since peeter is in the group “developers” the message is also added to “group:developer:messages” list
Since the group developers belongs to a watchlist (a group of groups) called “it”, the message is also added to "watchlist:it:messages" list

现在我们有了一个新的要求，列表需要附加过滤器。因此，组“开发者”将有一个过滤器“+php-javascript”，只显示来自该组中与该过滤器匹配的用户的消息

我们想把整个事情转移到elasticsearch。因此，我们将按以下格式为邮件编制索引：

{ 
    message : "PHP is awesome and Javascript is awesome and Java is awesome",
    user : "peeter",
    groups : ["developers", "architects"],
    watchlists : ["it", "tech personel", "weekend workers", "emergency staff"]
}

{ 
    query_string : "+php -javascript",
    term : {
        "groups" : "developers"
    }
}

{ 
    message : "PHP is awesome and Javascript is awesome and Java is awesome",
    user : "peeter"
}

{ 
    query_string : "+php -javascript",
    term : {
        "user" : "peeter", "michael", "jamie"
    }
}

当我们查找“开发者”组时，我们将以以下格式查询elastic：

{ 
    message : "PHP is awesome and Javascript is awesome and Java is awesome",
    user : "peeter",
    groups : ["developers", "architects"],
    watchlists : ["it", "tech personel", "weekend workers", "emergency staff"]
}

{ 
    query_string : "+php -javascript",
    term : {
        "groups" : "developers"
    }
}

{ 
    message : "PHP is awesome and Javascript is awesome and Java is awesome",
    user : "peeter"
}

{ 
    query_string : "+php -javascript",
    term : {
        "user" : "peeter", "michael", "jamie"
    }
}

问题是这些列表经常变化。新用户添加到“开发者”组，新组添加到观察列表。您还可以将两个组合并在一起，使“开发人员”和“架构师”成为 一个叫做“开发架构师”的小组。我们将留下一个索引，该索引在大量阅读时不断更新

我们的第二个想法是按以下格式为文档编制索引：

{ 
    message : "PHP is awesome and Javascript is awesome and Java is awesome",
    user : "peeter",
    groups : ["developers", "architects"],
    watchlists : ["it", "tech personel", "weekend workers", "emergency staff"]
}

{ 
    query_string : "+php -javascript",
    term : {
        "groups" : "developers"
    }
}

{ 
    message : "PHP is awesome and Javascript is awesome and Java is awesome",
    user : "peeter"
}

{ 
    query_string : "+php -javascript",
    term : {
        "user" : "peeter", "michael", "jamie"
    }
}

当我们查找“开发者”组时，我们将以以下格式查询elastic：

{ 
    message : "PHP is awesome and Javascript is awesome and Java is awesome",
    user : "peeter",
    groups : ["developers", "architects"],
    watchlists : ["it", "tech personel", "weekend workers", "emergency staff"]
}

{ 
    query_string : "+php -javascript",
    term : {
        "groups" : "developers"
    }
}

{ 
    message : "PHP is awesome and Javascript is awesome and Java is awesome",
    user : "peeter"
}

{ 
    query_string : "+php -javascript",
    term : {
        "user" : "peeter", "michael", "jamie"
    }
}

这种方法的问题是“开发者”组中可能有多达2000个用户

---

这两种解决方案是否都有效

提高查询属于“开发者”组的用户列表的性能的一种方法是使用术语查找机制-

因此，您可以有一个单独的索引（或类型），其中包含每个组的文档。在该文档中，您持有属于该组的用户列表。然后将术语查询指向该文档中的术语进行匹配

---

您的观察列表条款可能也可以这样做。

在ES的基础上构建的logstash（）是否已经提供了您所需的大部分/所有内容？这不是用于日志文件管理。是的，我意识到了这一点，但日志条目和事件消息在概念上是否相似？无论如何，在第二点上，尽管我不认为这是一个好的组合，因为你需要“观察名单”的概念。